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WORKSHOP  ON  AERONAUTICAL  DECISION  MAKING 


1.0  INTRODUCTION 

The  1992  FAA  sponsored  Aeronautical  Decision  Making  Workshop  was  a  part  of  a 
continuing  effort  to  enhance  safety  by  improving  the  operational  capabilities  of  pilots  and 
crews.  The  goal  of  the  workshop  was  to  bring  together  a  small  effective  group  of 
individuals  from  government,  industry  and  academe;  persons  who  were  actively  involved  in 
the  development  and  evaluation  of  decision  making  and  resource  management  materials, 
courseware,  simulator  scenarios,  or  other  products  suitable  for  incorporation  in  the  next 
generation  of  training.  This  concept,  as  well  as  the  workshop,  was  enthusiastically 
supported  by  the  participants  from  the  U.  S.  Air  Force,  the  U.  S.  Navy,  several  major 
airlines,  the  National  Aeronautics  and  Space  Administration,  the  Federal  Aviation 
Administration,  research  companies  and  the  University  of  Colorado.  The  workshop,  held 
in  Denver,  Colorado  on  May  6,7,  1992,  was  organized  and  conducted  by  Advanced  • 
Aviation  Concepts,  Inc.  as  a  part  of  FAA  contract  number  DTFA01-90-C-00042. 

This  report  is  provided  in  order  to  document  the  background  and  the  objectives,  as  well  as 
the  accomplishments,  of  the  workshop.  It  is  also  designed  to  provide  a  rudimentary 
understanding  of  the  proceedings  of  each  Working  Group  convened  at  the  workshop;  the 
specific  issues  discussed;  the  results  obtained;  and,  the  recommendations  provided.  One 
other  volume  will  be  provided  for  those  interested  in  a  more  complete  description  of  the 
background  research:  Volume  II  provides  the  Workshop  Presentations  and  the  technical 
recommendations  in  the  form  of  an  Action  Plan. 

It  is  important  to  note  that  the  workshop  addressed  the  overall  operations  concept  of 
Aeronautical  Decision  Making  as  an  integral  part  of  pilot/crew  training.  The  training 
identified  at  the  workshop  as  being  needed  immediately  was  derived  from  a  value 
assessment  and  the  realization  that  recognition  of  the  value  of  ADM  is  critical  to  acceptance 
of  the  training  by  both  pilots  and  management.  The  workshop  participants  identified  the 
behavior  and  attitude  elements  which  should  be  addressed  in  future  training  programs. 

Also,  the  diverse  needs  of  these  aviation  communities  were  addressed:  Federal  Aviation 
Regulations  (FAR)  Part  121,  Part  135  (for  both  airplanes  &  helicopters).  Part  91  (airplanes 
and  helicopters),  general  aviation  and  military. 

1.1  Objectives 

The  major  reason  for  convening  this  select  group  of  experts  on  ADM  was  to  share  ideas, 
identify  needs  and  explore  future  directions  for  research.  The  following  objectives  were 
established. 
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One  thrust  for  the  development  of  ADM  training  was  similar  to  that  for  Crew  Resource 
Management  (CRM):  A  well-publicized  series  of  fatal  crashes  which  were  attributed,  in 
part,  to  poor  crew  coordination/communication  and  judgment.  The  other  driver  for  ADM 
was,  is  and  will  be  the  need  to  provide  skills  through  training  which  can  be  an  initial 
substitute  for  those  gained  by  flight  experience.  The  reason  here  is  three-fold. 

1.  While  experience  does  teach,  the  cost  of  the  lesson  is  often  too  high— and,  the 
lesson  is  often  not  learned  until  after  the  experience.  Sadly,  some  pilots  are 
not  alive  after  the  experience.  As  has  been  said:  "Experience  is  not  a  fair 
teacher;  the  test  is  given  before  the  lesson  is  taught." 

2.  Also,  experience  can  teach  the  wrong  lesson,  or  teach  a  lesson  wrongly.  A 
poor  or  incorrect  decision  can  be  made  and  implemented  which  then  actually 
works  out  "for  all  the  wrong  reasons".  Since  it  turned  out  seemingly  well, 
the  likelihood  of  repeating,  or  trying  to  repeat,  this  poor  choice  and  • 
procedure  (in  a  similar  situation)  is  increased.  All  too  often,  the  result  is  an 
accident  or  incident. 

3.  It  has  been  found  that  there  are  levels  of  decisional  skills  and  expertise.  The 
characteristics  of  expert  cognitive  processes  have  been  defined  in  many 
diverse  fields  such  as  music,  medicine  and  athletics.  The  relationship 
between  expertise  and  training  or  practice  is  also  well  defined  in  these  fields. 
If  methods  could  be  found  to  teach  "expert"  pilot  decision  making  skills  to 
low  and  mid-time  pilots,  then  they  will  be  better  equipped  to  make  safe, 
effective  decisions  without  needing  to  await  high  levels  of  flight  time  and 
experience. 

However,  no  matter  how  one  looks  at  ADM,  the  bottom  line  is  a  need  to  equip  all  pilots, 
especially  the  low-timers,  with  better  ADM  skills  through  training. 

The  seminal  FA  A  study  (Jensen  and  Benel,  1977)  documented  the  need  for  ADM  training 
to  reduce  "pilot  error"  accidents.  The  authors  identified  the  need  for  (what  was  then 
called)  pilot  judgment  training  and  concluded  that  judgment  skills  could  indeed  be  taught. 
During  the  remainder  of  the  1970’s  and  through  most  of  the  1980’s,  materials  and 
programs  were  developed  in  pilot  judgment  training  (RDM);  many  of  these  had  R&D 
aspects.  As  a  result  of  this  work,  RDM  materials  became  available  to  the  aviation 
community  in  the  USA,  Canada  and  Australia.  These  materials  took  the  form  of  manuals 
published  by  Transport  Canada,  Australian  Department  of  Aviation.  Aircraft  Owners  and 
Pilots  Association,  and  the  Helicopter  Association  International.  Additionally,  a  series  of 
twelve  FAA  training  manuals  were  published  covering  such  topics  as  helicopter  ADM, 
Instrument  flight,  EMS  operations,  administrative  risk  management  and  more  (DOT/FAA, 
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1986-1991  inclusive). 


During  this  same  time  frame,  much,  if  not  all,  of  the  ADM  R&D  and  publications  were 
aimed  at  single  pilot,  general  and  corporate  aviation  operations.  FAR  Part  121  carriers 
were  developing  ADM  training  primarily  as  a  part  of  their  CRM  training  programs. 

In  retrospect,  this  era  resulted  in  quite  a  good  deal  of  research  and  materials  but  did  have 
some  lack  of  cooperative  efforts,  data  sharing  and  common  target  groups.  There  were  (and 
still  are)  some  basic  questions  which  needed  answers,  as  well  as  some  basic  differences  of 
opinion  on  what  constituted  ADM  and  its  place  in  a  training  curriculum. 

B.  The  1990’s 

ADM,  along  with  "situational  awareness",  has  again  become  a  topic  of  discussion  and 
limited  R&D.  Some  operational  personnel  and  research  personnel  share  the  view  that  there 
are  still  basics  (definitions,  concepts,  etc.)  which  either  are  not  resolved  or  are  totally 
unaddressed.  Additionally,  advances  in  CRM  have  begun  to  resurrect  some  old  concerns 
about  ADM  and  CRM.  These  CRM  advances,  in  concepts  and  in  measurement,  also  led  to 
a  feeling  among  some  people  that  an  ADM  breakthrough  was  now  possible.  Put 
idiomatically,  perhaps  we  finally  know  what  we  didn’t  know  and  now  can  proceed  from 
there. 

C.  ADM  and  CRM:  Ties.  Similarities.  Differences 

A  long-standing  question  has  been:  Is  ADM  part  of  CRM  or  a  result  of  CRM?  (or  vice 
versa!).  Is  a  good  crew  decision  the  result  of  CRM  skills,  if  indeed  not  a  major  goal  of 
CRM,  or  are  CRM  skills  a  sine  qua  non  for  a  good  crew  decision? 

These  two  questions  suggest  that  CRM  and  ADM  have  long  been  tied  together.  There  are 
successful  ADM  programs  and  successful  CRM  programs.  ADM  modules  are  often  parts 
of  CRM  training  and  ADM  (Leadership  and  Decision-Making)  aspects  are  called  out  as 
CRM  behavioral/performance  markers  in  the  FAA’s  CRM  Advisory  Circular  120-51  A. 
ADM,  more  than  CRM,  currently  needs  research  attention  and  development  work. 

One  basic  similarity  of  ADM  and  CRM  (R&D  and  training)  is  their  mutual  goal  of  a  safe, 
efficient  mission.  Another  similarity  is  that  ADM  is  also  poised  to  make  major  advances, 
due  in  part  to  advances  in  CRM.  The  five  primary  cockpit  management  tools  which  have 
been  taught  in  both  ADM  and  CRM:  attention,  attitude,  crew  management,  risk 
management  and  stress  management  will  expedite  the  achievement  of  these  major  advances. 
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The  basic  difference  between  ADM  and  CRM  is  that  CRM  has  already  taken  some 
significant  strides  in  the  early  90’ s;  strides  in  identification  and  assessment  of  discrete  skills 
and  behaviors  which  make  up  "good"  CRM,  and  in  developing  a  3-tier,  or  phased  model  of 
training:  awareness,  indoctrination  and  recurrency.  (Helmreich  &  Foushee,  1986, 
Helmreich  &  Wilhelm,  1990).  On  the  other  hand,  ADM  is  now  in  a  position  to  take  new 
strides  and  make  advances  --  and  ADM  needs  to  make  these  advances.  In  short,  ADM 
needs  work,  through  new  R&D. 

The  question  of  whether  CRM  is  part  of  ADM,  or  the  reverse,  now  seems  academic.  Both 
are  vital  components  of  mission  safety  and  mission  performance;  both  offer  vital 
information  and  understanding  to  the  other;  both  must  continue  to  be  stressed,  expanded 
and  refined;  and,  both  must  become  transparently  embedded  in  flight  trai  ling  and  openly 
integrated  with  the  technical  (flight  control)  skills  of  which  they  are  truly  a  part. 

[Aside:  A  major  area  where  CRM  and  ADM  can  immediately  work  together  is  in  the 
development  of  better  (LOFT)  scenarios.  Improved  LOFTs  can  be  developed  based  on 
focused  training  requirements  for  CRM,  ADM,  normal/abnormal/emergency  procedures, 
motor  skills,  etc.  Better  understanding  and  use  of  ADM  events  in  a  LOFT  can  enable 
better  scenario  design  and  better  pilot/crew  assessment.] 


2.0  THE  ISSUES 

A.  ADM  has  not  yet  developed  a  well-defined  and  comprehensive  set  of  constructs  and 
a  theory  of  its  own  (Maher,  1991).  Rather,  general  decision-making  theory  was  applied 
and/or  modified  for  ADM.  The  results  were  predictable:  a  simplification  process  whereby 
a  linear  model  (e.g.,  PASS,  SAFE,  DECIDE)  was  used.  Such  models  are  time-consuming, 
cumbersome  and  inadequate  to  the  dynamic,  often  time-compressed,  decisional  situations 
that  aviators  face  when  flying.  In  point  of  fact,  it  seems  fairly  accurate  to  say  if  a  linear 
model  was  taught  on  the  ground,  it  was  rarely  used  aloft. 

Part  of  this  issue  was  that  most  decision-making  theories  began  with  how  or  why  people 
make  errors.  A1  to  often  the  result  of  the  decision-making  process  was  to  prevent  errors, 
especially  catastrophic  ones.  However,  the  problem  of  how  to  identify  the  elements  of  a 
good  decision,  and  of  how  to  enhance  both  the  capability  and  the  probability  of  making  one 
were  not  adequately  addressed.  There  was  an  assumption  that  error  elimination  was 
equivalent  to  good  decision-making.  A  major  player  in  this  school  of  thought  was  the 
linear,  causal  chain  of  event  concept  of  accidents,  where  the  goal  is  breaking  the  causal 
chain  and  the  assumption  is  that  breaking  the  chain  precluded  an  accident  (Diehl,  1990). 
New  findings  on  the  multi-dimensional,  multi-"layered"  processes  which  people  actually  use 
to  make  decisions  --  often  projecting  forward  and  backward  in  time  in  their  thinking  —  have 
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shown  an  insufficiency  in  the  linear  models  of  decision-making.  A  view  that  ADM  was  too 
complex,  contained  too  many  variables,  and  was  too  dynamic  for  any  one  model  to  capture 
also  existed.  An  offshoot  of  this  view  was  that,  while  no  training  would  be  truly  complete 
and  adequate,  any  training  was  better  than  none  at  all.  This  view  was  complementary  to 
the  use  of  linear  models  and  reinforced  teaching  them. 

B.  Very  recently,  several  new  models  and  concepts  of  how  people  actually  make 
decisions,  rather  than  how  they  make  errors,  have  been  developed  (Klein  &  Klinger,  1991, 
Orasanu  &  Salas,  Orasanu).  Another  advance  has  been  in  the  area  of  team  decisional 
processes,  especially  in  naturalistic  (i.e.,  on  the  job)  settings,  rather  than  in  an  artificial 
laboratory-like  experimental  setting. 

C.  The  final  issue,  or  perhaps  driver,  is  the  new  efforts  which  explore  Expert  Decision 
Making  (EDM)  in  operational  settings  by  researchers  in  and  out  of  the  Government  (Adams 
&  Ericsson,  1992,  Ericsson  &  Smith,  1991,  Bloomfield,  1992,  Bloomfield,  Edens,  e(.  al., 
1992)  and,  to  a  lesser  degree,  by  some  major  air  carriers. 

The  sum  of  A,B,C  (above)  was  the  authors’  realization  that  a  series  of  ADM  workshops, 
bringing  together  the  operational  "users",  the  researchers  and  the  FA  A  was  timely  and 
appropriate. 


3.0  THE  NEED  FOR  A  WORKSHOP 

A.  The  rationale  is  simple  enough  and  is  actually  contained  in  the  two  preceding 
sections.  Historically,  the  time  was  right  to  begin  new,  coordinated  ADM  R«feD  work. 

First  there  were  new  findings,  new  interests,  new  needs  and  basic  issues  which  still  had  not 
been  answered  and  which  needed  to  be  resolved.  Second,  there  was  a  growing  realization 
that  ADM  was  in  real  need  of  new  models  and  materials  which  accurately  identified, 
defined,  and  enhanced  ADM  skills.  Third,  the  past  two  years  of  FAA-sponsored  research 
on  the  cognitive  processes  of  expert  pilots  has  reached  a  level  of  maturity  which  suggested 
that  a  broader  exposure  and  use  was  possible. 

B.  There  were  also  certain  things  which  had  occurred  in  CRM  and  other  aviation  R&D 
areas  which  the  ADM  workshop  sought  not  only  to  correct  but  also  to  preclude  in  the 
future.  The  workshop  was  designed  to  bring  together  a  diverse  group  of  experts  who  were 
doing  or  had  done  work  in  decision  making,  from  a  variety  of  perspectives.  In  this  way, 
there  would  be  an  initial  cross-pollination  which  could  be  built  upon  and  expanded  in  later 
workshops  and  projects.  The  workshop  also  was  an  effort  to  preclude  a  premature 
"standardization"  of  ADM  at  a  time  when  new  concepts  and  development,  not  codification, 
are  needed.  ADM  needs  to  be  an  open  and  expanding  discipline,  not  the  domain  of  any 
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"in-group".  The  workshop  was  a  step  toward  opening  ADM  to  all  by  evaluating  the  merits 
of  their  concepts  on  scientific,  logical  grounds. 


4.0  THE  PROCESS 

A.  Preparation 

The  workshop  was  conceived  in  mid-1991  because  the  authors’  interest  and  work  in  ADM, 
input  from  aircrew  training  officers  at  several  major  air  carriers,  and  with  coordination  with 
NASA,  private  and  academic  researchers. 

During  February  1992,  an  initial  list  of  participants  and  a  preliminary  set  of  objectives  and 
agenda  was  prepared.  Each  potential  participant  was  mailed  the  rough-cut  objectives  and 
draft  agenda,  as  well  as  a  cover  letter  and  other  materials.  This  package  solicited  the 
participant’s  preferences  for  possible  meeting  sites  and  offered  a  tentative  May  time-frame. 
The  personnel  receiving  this  package  were  encouraged  to  indicate  their  willingness  to 
attend/present,  and  their  choice  of  workshop  site  was  solicited. 

Based  on  the  responses  received  to  this  initial  mailing,  a  second  letter  was  prepared,  some 
new  participants  targeted,  and  others  deleted  [by  their  choice],  from  the  initial  roster.  It 
should  be  noted  that,  throughout  this  process  of  planning  and  arranging  the  workshop,  both 
Rich  Adams  and  Ron  Lofaro  had  numerous  telephone  conversations  with  the 
proposed/actual  participants,  explaining  details  and  goals. 

In  April,  a  final  package  containing  a  revised  set  of  objectives,  agenda,  requests  for 
biographies  and  hard  copies  of  their  presentations,  choice  of  conference  site.  The  details  on 
travel  and  hotel  accommodations  also  were  mailed  to  the  participants  at  this  time. 

Advanced  Aviation  Concepts,  Inc.  prepared  a  workshop  workbook  which  was  given  to  the 
participants  on  Day  I.  Major  features  of  the  workbook  were  a  brief  resume  of  the 
presenters  and  the  use  of  formatted  areas  where  notes  and  questions  concerning  each 
briefing  could  be  recorded  and  later  used  in  group  discussions  as  well  as  referenced  after 
the  conference.  It  included  papers  on  EDM  (Adams  and  Lofaro,  1992),  as  well  as  on  CRM 
(Lofaro,  1992).  The  same  workbook  format  was  included  to  aid  the  participants  in 
recording  working  group  discussions,  conclusions  and  recommendations.  A  sample  copy 
(edited  for  space  requirements)  of  this  workbook  along  with  pre-defined  goals  and 
objectives  is  provided  in  Appendix  A. 
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B.  Day  I 


Ron  Lofaro  welcomed  the  20  participants  and  briefly  discussed  ADM  ("How  it  got  to  be 
where  it  is"),  the  rationale  for  the  Workshop,  and  provided  a  brief  discussion  of  the  variety 
of  approaches  to  CRM/ADM,  simulator  usage  and  the  existence  of  substantial  experience 
which  has  produced  the  following  set  of  programs,  regulations  and  advisory  guidelines. 

1.  Existing  Training  Programs: 

a.  LOS  -  Line  Oriented  Simulation 

b.  LOFT  -  Line  Oriented  Flight  Training 

2.  AQP  -  Advanced  Qualification  Program  (SFAR‘  58) 

3.  CRM  -  Crew  Resource  Management  (new  AC  120-51  A) 

4.  ADM  -  Aeronautical  Decision  Making  (AC  60-22) 

Rich  Adams  of  AAC  then  reviewed  the  objectives,  processes  and  organization  of  the 
Workshop.  For  the  remainder  of  the  day  (0830  -  1730),  nine  (9)  people  made 
presentations,  with  Q&A  sessions  immediately  following  each  presentation.  It  was  a  mark 
of  the  skill  of  the  presenters  and  the  relevancy  of  their  presentations  that  this  very  long  first 
day  did  not  have  any  "drop-outs"  either  by  any  presenter  or  because  of  lack  of  sufficient 
time.  It  is  a  more  significant  mark  that,  while  the  participants  all  agreed  it  was  a  long  day, 
they  also  agreed  it  was  a  day  well  spent  and  they  had  retained  their  interest  throughout. 

The  high  levels  of  interest  were  further  reflected  in  an  informal  "mixer",  wherein  further 
questions  and  discussions  flowed  for  over  two  hours. 

C.  Day  n 

There  was  one  final  presenter.  After  this  presentation,  Ron  Lofaro  reviewed  the  working 
group  objectives;  Rich  Adams  dealt  with  the  structure  of  each  work  group  and  the  proposed 
outputs.  The  working  groups  then  convened  in  individual,  separate  work  areas.  After  some 
3  1/2  hours,  all  participants  re-assembled  in  the  main  conference  room  to  hear  the  3 
working  groups  present  their  discussions  and  recommendations. 


’  SPAR  -  Special  Federal  Aviation  Regulation 
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After  this  time  was  given  to  an  examination  of  similarities  and  differences  across  the 
working  groups  and  a  short  discussion  of  possible  actions  and  projects  in  ADM.  Ron 
Lofaro  and  Rich  Adams  did  a  brief  processing-out  and  delivered  closing  comments  and 
thanks. 

Again,  a  benchmark  of  the  level  of  participant  interest  and  involvement  was  that,  after 
another  long  day  and  after  the  workshop  had  been  closed,  the  participants  remained  to 
discuss  issues  and  plans  informally  for  well  over  another  hour. 


5.0  KEY  CONCEPTS  AM)  RECOMMENDATIONS 

This  workshop  took  ADM  several  steps  forward,  produced  several  seminal  concepts  and 
one  major  re-look/. e-questioning  at  ADM.  Appendix  B  provides  a  summary  of  the  specific 
technical  contributions  of  the  eleven  presenters  at  the  workshop.  There  was  an 
extraordinary  similarity  among  the  issues,  discussions,  and  conclusions  across  the  3  work 
groups  even  though  one  purpose  of  the  3  work  groups  was  to  provide  the  structure  for 
diversity  and  differentiation. 

The  working  groups  had  different  foci:  Future  Civil  Aviation  Research  &  Training;  Future 
Military  Aviation  Research  and  Training;  and,  Advanced  Training  &  Evaluation  Methods. 
Each  work  group  analyzed  ADM  problems  and  needs  within  its  area  of  emphasis. 

However,  the  commonality  of  concern  and  conclusions  across  the  groups  is  a  significant 
indicator  of  a  generic  quality  to  ADM,  a  set  of  similarities  which  exist  in  all  aviation  flight 
operations,  be  it  military  private  or  commercial.  At  the  same  time,  ADM  also  was  seen  as 
very  specific  to,  and  conditional  upon,  both  the  type  of  aircraft  and  the  mission  being 
flown.  The  outputs  of  the  working  groups  are  presented  in  Sections  6.0,  7.0,  and  8.0;  these 
contain  fairly  complete  records  of  the  discussion  topics  and  results  of  the  deliberations  for 
each  group. 

The  general  conclusions  reached  by  the  participants,  both  in  their  work  groups  and  in  the 
intact,  entire  group  can  be  briefly  summarized  by  these  statements. 

A.  ADM,  like  CRM,  must  be  embedded  in  all  levels  of  an  organization.  Without  real 
"buy-in"  and  training  for  all  organizational  units,  ADM  will  not  have  the  operational 
impact  it  can  engender:  increased  safety  and  increased  crew  performance  and  effectiveness. 

B.  This  workshop  should  be  the  first  step  in  a  new  awareness  period  for  aviation  of  the 
status,  promise,  prior  work  and  future  efforts/goals  of  ADM  and  EDM. 

C.  This  workshop  demonstrated  a  truly  remarkable  convergence  and  "match"  between 
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the  operational  community  and  the  R&D  community  as  to  the  status,  goals  and  needs  of 
ADM. 

D.  There  already  is  some  application  and  extension  of  such  research-generated  concepts 
and  techniques  as  Naturalistic  Team  Decision-Making  and  Expert  Decision-Making  in  the 
Aviation  arena. 

E.  There  is  a  major  question  to  be  answered,  which  requires  an  accurate  re-look  and 
re-definition:  "What  is  a  real  aircrew/pilot  decision?"  —  that  is  to  say,  when  does  an  event 
generate  a  true  decisional  opportunity  for  a  pilot/crew  versus  a  "one-path  only"  reaction, 
where  the  real  emphasis  is  not  on  decisional  aspects  but  procedures.  A  new  term  for  this 
"real"  decision  was  suggested,  "operational  decision";  one  which  places  decisional  demands 
on  the  flight  crew  and  flight  system  to  resolve  an  ambiguous  situation. 

F.  Stemming  from,  and  interlocked  with  (4)  above,  was  an  initial  proposal  that  3 
classes  of  aeronautical  decisions  exist.  These  are: 

1 .  Binary  Decision  Making:  A  simple  "go/no  go"  call,  one  where  there  are 

only  two  paths  open.  This  was  referred  to  as  somewhat  of  a  lower  powered 
cognitive  task  where  the  actual  issues  were  timely  response  and  using  correct 
technique  in  a  situation  which  actually  had  only  one  correct  response. 
"Correct"  here  is  used  in  the  sense  of  the  alternate  response  leading  to  an 
accident  or  incident;  a  response  which  could  only  lead  to  failure.  The  level 
of  cognitive  processing  required  to  make  the  decision  is  the  main 
discriminator.  The  binary  decisions  are  often  made  at  such  a  low  level  that 
the  person  making  the  decision  is  virtually  unaware  a  decision  has  been 
made.  There  is  a  high  level  of  automaticity,  due  to  training,  to  the 
pilot/crew  response  in  this  class  of  decisions. 

This  type  of  decision  may  be  typified  best  by  an  engine  out  at  V,.  The  issue 
here  is  not  so  much  what  to  do— the  pilot/crew  are  trained  to  respond  to  this 
event-but  the  timely,  accurate,  skillful  application  of  the  techniques 
necessary  for  rotation  and  an  immediate  landing  at  the  airport;  along  with 
any  other  procedures  required  by  the  terrain  and  weather  for  accomplishing 
the  safe  landing.  This  type  of  decision  represents  a  simple  cognitive  effort 
and  requires  less  time  than  the  more  complex  classes  (B.  and  C.  below) 
which  may  require  significant  cognitive  workload  and  processing  time. 
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2.  Rule  Based  decision  making:  A  decisional  class  where  there  must  be  some 
planning  and  problem-solving.  In  this  class,  the  decision  was  more  complex, 
but  still  could  end  up  as  a  "go/no  go"  type.  There  still  was  some  emphasis 
on  accurate  and  quick  recognition  with  the  response  to  be  mainly  the 
application  of  correct  procedures. 

However,  in  this  class  of  decisions,  the  complexity  of  the  environment  and 
conditions  led  to  more  actual  problem-solving  and  decisional  strategies  than 
in  the  first  class. 

3.  Adaptive  decision  making:  The  third  type  of  decision  has  so  much 
ambiguity  embedded  in  it— conditions,  causes,  differing  possible  courses  of 
action— that  it  requires  all  the  skills  needed  for  classes  1  and  2,  plus  a  high- 
level  of  diagnostic  skill  and  team  work.  This  type  of  ADM  is  best  typified 
by  United  Flights’  232  and  811.  It  is  this  type  of  decision  which  needs  new 
emphasis  and  needs  the  most  supporting  R&D  to  identify,  understand,  and 
integrate  the  condition  sets,  operational  demands,  and  decisional  strategies 
with  successful  outcome  behaviors  and  performances. 

NOTE:  The  participants  recognized  that  overlap  exists  among  these  three  aeronautical 
decision  classes.  A  major  R&D  effort  would  be  to  define/refine  all  of  these  three  classes. 

G.  Again,  stemming  from  and  interlocked  with  both  (4)  and  (5)  is  the  consensus  that,  in 
order  to  do  (4)  and  (5),  several  R&D  products  are  absolutely  essential.  These  are: 

1.  Accomplishing  what  was  termed  a  "cognitive  task  analysis"  (CTA),  broken 
out  by  phases  of  flight  in  a  mission.  This  analysis  must  embed  the  cues, 
conditions,  environment  and  mission  goals. 

2.  Developing  a  taxonomy  and  classification  schema  of  decisional  events  (this 
will  define  what  is  "real"  aeronautical  decision  making)  and  decisional 
strategies. 

3.  Identifying  and  articulating  how  to  use  the  Cognitive  Task  Analysis  and 
Decision  Making  Taxonomy  in  the  development  of  pilot/crew  training,  and 
especially  in  the  development  of  expert  LOFT  training  scenarios  based  on 
actual  decisional  events. 
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H.  Finally,  there  was  the  strong  recognition  that  ADM  R&D  and  strategies,  while 
having  similar  aspects  and  components,  were  differentiated  by  aircraft  type  and  mission.  It 
was  hoped  that  generic  ADM  materials,  methods  and  tools  could  be  developed  as  well  as 
paradigms  for  extending  generic  findings/components  to  specific  aircraft  and  missions. 


Let  us  look  once  again  at  the  three  proposed  classes  of  decisions.  There  was  a  group 
recognition  that  not  only  are  there  three  classes  of  crew/pilot  decisions,  but  that  differential 
conditions—such  as  mission  (long;  medium;  short-haul)  and  aircraft  type  (automated  versus 
non-automated)  lead  to  different  decisional  needs  and  processes.  Cues  (availability,  type, 
recognition)  and  conditions  (context,  environment,  phase  of  flight),  along  with  time 
pressures,  are  the  drivers  for  all  types  of  decisional  processes.  The  type  of  flight 
operations  (normal,  abnormal,  emergency)  is  significantly  related  to  the  class  of  decision. 

In  the  first  type,  we  are  usually  looking  at  normal  operations,  with  the  emphasis  on 
procedures,  sequencing  of  responses  and  prioritiiation/timing.  While  not  quite  as  simple  as 
a  "go/no  go",  this  first  class  of  decision  has  fewer  crew  decisional  needs  for  cues.  It  also 
has  fewer  conditions. 

The  second  class  of  decision  has  multiple  alternatives  which  are  usually  known.  Time,  not 
timing,  is  a  factor  and  the  thrust  is  on  finding  the  right  procedure(s)  and  implementing  it 
rapidly  and  precisely.  These  are  usually  abnormal  operations.  If  the  decision  maker  has 
learned  the  procedures,  he/she  will  know  what  to  do  after  reaching  a  level  of  situational 
awareness.  The  ambiguity  lies  in  assessing  the  situation.  This  is  particularly  true  in  time- 
compressed  situations.  The  final  decisional  type  features  ambiguity  and  previously  un¬ 
encountered  circumstances,  where  all  aspects  of  the  first  two  types  are  present  and  the 
focus  is  on  the  diagnostic  skills  needed  to  use  the  cues  and  conditions,  usually  in  a  time- 
compressed  situation.  This  is  usually  an  emergency  situation.  These  circumstances  require 
the  expert’s  ability  to  recognize  and  combine  patterns  from  previous  individual,  related 
situations,  reason  forward  and  backward  to  come  up  with  a  new  procedure  or  plan  while 
continually  monitoring  the  performance  and  outcome  of  the  plan.  A  key  point  is  that  both 
class  2  and  3  decisions  can  be  "reduced",  i.e.,  the  decisional  skills  and  processes  of  the 
crew/pilot,  can  make  the  decisions  abnormal  or,  in  some  cases,  normal  operational 
decisions. 
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6.0  WORKING  GROUP  A:  FUTURE  CIVIL  AVIATION  RESEARCH  & 
TRAINING 

This  group  was  tasked  with  analyzing  the  status,  accomplishments  and  shortcomings  of 
ADM  in  the  civil  aviation  environment.  Emphasis  was  placed  on  the  commercial  service 
training  of  Part  121  and  Part  135  pilots  and  crews.  The  group  chairman  was  Dr.  Ronald 
John  Lofaro  of  the  FAA’s  Research  and  Development  Service,  Systems  Technology 
Division,  ARD-2(X)  (202)  267-8529.  The  industry  co-chair  was  Catherine  A.  Adams  of 
Advanced  Aviation  Concepts,  Inc.  (407)  747-3414.  The  subject  matter  experts  and 
technical  support  staff  were; 

Mr.  Jan  Demuth 
Dr.  David  Hunter 
Dr.  John  Bloomfield 
Dr.  Gary  A.  Klein 
Mr.  James  E.  Irving 

6.1  Introduction 

The  group  discussion  was  based  on  the  identification  of  problems  and  immediate  needs  in 
ADM  and  some  effort  at  finding  paths  to  resolving  the  problems.  Initial  group  discussion 
centered  around  what  was  perceived  as  barriers  to  new  ADM  training  being  implemented 
by  major  air  carriers  and  various  ways  to  approach  this.  This  initial  discussion  lead  to  the 
following  points  and  concepts,  built  around  this  3-part  premise: 

Air  carriers  will  see  any  "new"  training  as  adding  to  the  overall  cost  of,  and  subtracting 
from  the  bottom-line,  profit.  What,  then,  can  be  done  to  combat  and  nullify  this?  ADM 
training  must  not  be  seen,  presented,  or  become  another  add-on  piece  of  training:  as  a 
band-aid  for  some  set  of  problems.  Rather  it  must  become  an  integral,  necessary  part  of  a 
comprehensive  successful  program.  Suggestions  were  made  during  the  dialogue  of  the 
participants  in  light  of  the  premise. 

A.  ADM,  perhaps,  can  be  integrated  into  flight  training,  as  the  FAA’s  AQP  (Advanced 
Qualification  Program)  SFAR  58  (Special  Federal  Aviation  Regulation  number  58)  calls  for 
CRM  to  be  integrated  into  the  technical  aspects  of  flight  training. 

B.  ADM  must  be  transparent  -  it  must  be  so  finely  interwoven  in  the  fabric  of  training 
that  it  is  not  perceived  as  add-on  material  or  a  separate  module. 

C.  ADM  training  for  airline  "new  hires"  may  be  the  initial  thrust,  followed  by  making 
ADM  a  part  of  a  pilot’s  career  education  progression- "a  career-long  event". 


FAA  Air  Carrier  Training 
FAA  Aviation  Medicine 
Honeywell  S.  R.  C. 

Klein  Associates,  Inc. 
Captain,  United  Airlines 


(202)  267-8922 
(202)  366-6935 
(612)  782-7674 
(513)  873-8166 
(303)  444-9467 
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D.  Senior  airline  captains  should  be  key  participants  and  called  upon  to  formulate  parts 
of  this  initial  thrust,  giving  a  simultaneous  top-down  and  bottom-up  approach. 

E.  An  "ADM  awareness"  program,  both  within  air  carriers  and  in  the  aviation 
community,  needs  to  be  started  to  develop  an  awareness  of  the  value  and  benefits  of  ADM 
to  pilots  and  management.  (This  is  not  to  be  confused  with  the  Awareness  Phase  of  CRM 
training.) 

F.  ADM  can  be  introduced  by  using  it  as  an  analytic  tool  on  operational  procedures,  by 
and  across  fleets.  For  example,  analysis  and  discussion  of  decisional  strategies  and  styles 
in  current  situations  or  scenarios  could  establish  linkages  to  training  and  training  needs. 
This,  then  provides  a  rationale  for  the  development  and  inclusion  of  ADM  in  flightcrew 
training. 

6.2  Problems  and  Needs 

There  were  two  major  problem  (need)-areas  agreed  on  by  this  workgroup.  Both  of  these 
are  contained  in  the  previous  portion  of  the  Key  Concepts  and  Recommendations  section. 
However,  it  may  be  instructive  to  state  these  as  the  working  group,  rather  than  the  entire 
Workshop,  formulated  them. 

The  first  issue  was  the  immediate  need  to  begin  R&D  on  a  common  and  basic  classification 
schema  with  a  taxonomy  for  aeronautical  decision-making  with  validation.  This  could  be 
developed  in  part  from  the  initial  work  in  #6  above.  Parallel  and  interlocked  with  this 
effort  was  the  need  for  (what  was  termed)  a  cognitive  task  analysis  (CTA).  In  this  CTA, 
attention  should  be  given  to  classes  of  decisions;  node  points  in  an  event  where  information 
is  absolutely  needed  by  the  crew;  identification  of  what  information  is  needed;  specification 
of  the  rationale  for  what  information  management  strategies  would  be  effective.  Embedded 
in  the  task  analysis  and  taxonomy  work  would  be  an  identification  of  where  the  operational 
impact  of  R&D  work  would  occur— training  design,  materials  and  procedures;  SOP’s;  other. 


Finally,  the  CTA  needs  to  provide  a  paradigm  for  further,  differential  CTAs  -  CTAs  which 
are  to  be  aircraft  specific  and  can  drive  specific  areas  of  training  development. 
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6.3  Future  Requirements  and  Recommendations 

The  two  problem  areas  addressed  by  this  group  define  the  need  for  basic  research  in  the 
ADM  domain.  For  example,  can  we  define  what  constitutes  a  "good  decision"  and  can  we 
identify  who  good  decision  makers  are?  The  thrust  of  the  group’s  recommendation  in  these 
basic  areas  was  to  examine  these  questions  in  the  context  of  the  operational  cognitive  task 
demands  for  each  flight  phase. 

The  recommended  form  and  substance  for  this  analysis  included: 

A.  Cognitive  Task  Analysis  &  Evaluation  for  normal,  emergency  and  abnormal 
procedures 

B.  Simulator  scenario  generation 

C.  Subject  pilot  test  and  evaluation 

D.  Debrief  and  re-analysis  of  cognitive  nodes  and  decisions  (good  vs.  bad)  and  decision 

maker’s  performance  (good  vs.  bad  qualities) 

Due  to  the  sophistication,  cost  and  complexity  of  this  research,  the  group  suggested  a  joint 
govemment/industry  effort  with  the  FAA  and  NASA  developing  the  program  goals  and 
objectives,  providing  technical  direction  and  monitoring  outputs.  R  &  D  support  from  the 
airlines,  and  researchers  from  both  industry  and  academe  would  be  used  to  spread  the 
workload  and  expedite  the  data  collection  and  interpretation  from  an  operational 
perspective. 
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7.0  WORKING  GROUP  B:  MILITARY  AVIATION  RESEARCH  &  TRAINING 


This  group  was  tasked  with  analyzing  the  similarities  and  differences  between  ADM  in  the 
military  and  civil  aviation  environments.  Emphasis  was  placed  on  the  analysis  of  ADM  in 
high  and  low  workload  military  missions,  relating  decision  making  training  needs  to  specific 
tasks  in  those  environments,  and  defining  training  strategies  applicable  to  those  tasks  in 
both  military  and  civil  aviation.  The  group  chairman  was  Dr.  Alan  E.  Diehl  of  the  USAF 
Safety  Agency  (714)  382-3458.  The  co-chair  was  Sharon  Irving  of  the  University  of 
Colorado  at  Boulder  (303)  444-9467.  Subject  matter  experts  and  technical  support  staff 
were: 

Dr.  Robert  A.  Alkov 
Dr.  Carolyn  Prince 
Dr.  Reid  Hastie 
Dr.  George  Kaempf 

7.1  Introduction 

A.  The  Military  Aviation  working  group  based  their  discussions  on  the  following 
observations. 

1 .  The  military  has  these  unique  capabilities: 

a.  access  to  large  quantities  of  data  and  information  not  otherwise 
available. 

b.  LOFT  scenarios  tied  to  demographic  data. 

c.  a  captive  group  of  subjects  (time  and  cooperation  is  insured). 

d.  a  highly  structured,  controlled  training  process. 

2.  Although  civilian  aviation  has  been  very  active  and  collected  a  great  deal  of 
ADM/CRM  data  using  airline  pilot/crew  volunteers,  the  results  obtained  may 
not  be  completely  representative  of  "typical"  line  pilots  due  to  the 
predisposition  of  the  volunteers  to  support  and  use  ADM/CRM. 

3.  Accident  events  (because  of  the  relative  infrequency  and  often  missing  data) 
are  not  always  the  best  source  of  information  for  training  purposes. 
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4.  The  military  training  and  laboratory  environments  can  often  provide  more 
controlled  test  situations, 

5.  There  is  a  safety-driven  need  to  determine  a  way  for  the  FAA’s  National 
Plan  for  Human  Factors  to  utilize  these  resources. 

6.  The  tactical  and  mission  requirements  of  military  flying  provide  an  extremely 
rich  environment  for  studying  pilot  decision  making  under  time-compressed 
and  stressful  conditions. 

B.  Using  these  basic  observations,  the  group  set  out  to  develop  an  analysis  of  future 
ADM  research  and  training  requirements  based  on  military  needs  but  with  generalized 
applicability  to  the  civil  environment.  They  wanted  to  avoid  spending  several  man-hours 
(actually  about  a  combined  half-manweek  was  available)  creating  a  wish  list  of  dream 
projects.  Therefore,  the  problems  were  framed  in  a  systematic  approach  to  develop  specific 
training  needs.  The  military  helicopter  and  transport  operations  environments  vvere  selected 
to  represent  high  and  low  decision  making  environments  respectively. 

The  group  used  the  information  from  the  presentations  on  the  first  day  of  the  workshop  (see 
Volume  I  Appendix  B  and  Volume  II  for  presentation  material  indicated  in  parenthesis)  as  a 
foundation  and  frame  of  reference  in  the  following  manner: 

1.  The  ADM  Effectiveness  Overview  (Diehl)  and  Trans-Cockpit  Authority 
Gradient  (Alkov)  discussions  were  used  to  formulate  the  macro  environment. 

2.  The  Recognition  Primed  (Klein)  and  Natural  (Bloomfield)  decision  making 
models  provided  two  theoretical  methods  for  analyzing  the  process  occurring 
in  the  cockpit. 

3.  The  transition  between  the  theoretical  and  the  operational  was  based  upon  the 
Bootstrapping  Experience  (Irving),  Shared  Mental  Models  (Orasanu)  and 
How  Experts  Think  (Adams)  presentations. 

4.  The  springboard  for  developing  advanced  training  were  the  presentations  on 
Decision  Making  in  Operational  Systems  (Smith)  and  the  Methodology  for 
Training  Expertise  (Ericsson). 

C.  The  group  concentrated  their  work  in  these  four  areas. 

1 .  Identifying  and  defining  decision  making  tasks  (The  work  group  felt  that 
aeronautical  decision  making  is  not  yet  clearly  understood  or  defined) 
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2.  Determining  Training  Objectives 

3.  Developing  Training  Strategies 

4.  Evaluating  Training  Effectiveness 

7.2  Problems  and  Needs 

The  approach  selected  by  this  group  recognized  that  there  is  a  great  deal  of  basic  research 
that  is  necessary,  including:  defining  the  concept  of  ADM  and  distinguishing  it  from  other 
concepts;  classifying  decisions;  learning  how  crews  deal  effectively  with  decisions; 
determining  how  this  can  be  trained;  and  evaluating  the  training  itself.  This  approach 
resulted  in  the  identification  of  four  problem  areas  requiring  future  R&D  work.  It  also  led 
the  group  to  an  examination  of  not  only  what  needed  to  be  done,  but  also  who  would  be 
affected,  who  should  be  tasked  to  perform  the  necessary  research  or  training  development, 
and  the  time  frame  within  which  it  might  be  targeted  for  completion.  The  following  pages 
present  a  brief  summary  of  the  analyses  performed  and  the  products  anticipated. 

A.  Identifying  and  defining  decision  making  tasks 

The  goal  in  this  area  is  to  identify  decision  tasks  and  how  decisions  are  made  in  as  many 
environments  (civil,  military  and  general  aviation)  and  as  many  mission  types  as  would  be 
practical  and  meaningful.  The  problem  statement  is  to  determine  the  various  methods  and 
models  that  could  be  used  to  identify  decision  making  tasks.  The  work  required  would  be 
to  identify  how  the  decisions  are  made,  the  cues  used,  the  goals,  the  expectancies,  etc. 

Once  we  know  how  experts  do  it,  then  we  can  develop  training  objectives  for  the  novices. 
The  work  required  would  explicitly  include  review  of  existing  models  such  as  those 
presented  by  Bloomfield,  Klein,  Orasanu  and  Smith  for  applicability.  Further  work  would 
include  the  specification  of  the  need  for,  and  possibly  the  types  of,  models  that  may  need  to 
be  generated  in  the  future. 

The  products  of  this  analysis  of  decision  tasks  and  applicable  methods  of  decision  making 
for  both  high  and  low  workload  tasks  would  be  a  specification  of  a  generalized 
methodology  and  documentation  of  the  taxonomy  of  decision  making  tasks  across  different 
environments  or  mission  types.  The  methodology  would  be  suitable  for  use  both  by 
training  departments  and  by  pilots  in  their  day-to-day  decision  making.  The  applicability  of 
these  results  would  be  across  the  user  communities  and  include  a  range  of  aircraft  and 
mission  types. 

The  team  proposed  to  accomplish  this  work  would  include  the  military  and  NASA 
laboratories/simulator  facilities  augmented  by  academic  and  industry  subject  matter  experts 
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and  working  under  the  supervision  of  an  FAA  Project  Manager  and  Technical  Monitor. 

B.  Determining  Training  Objectives 

The  problem  statement  in  this  area  is  to  address  the  specific  decision  making  training  needs 
for  each  aircraft  type  or  fleet.  What  the  military  has  to  offer  in  this  area  is  diversity  in 
crew  experience,  in  crew  size,  in  crew  tasking,  and  in  some  of  the  decisions  they  face. 

The  target  groups  postulated  at  this  time  were  air  transport  operations  and  helicopter 
operations  training.  What  the  military  also  has  to  offer  is  more  access  to  crews  for 
research  than  is  possible  in  the  airline  communities,  and  its  on-going  training  programs  at 
multiple  levels  that  provide  research  opportunities.  The  spectrum  of  pilot  decision  making 
training  needs  from  ab  initio  and  undergraduate  through  recurrent  training  could  be 
analyzed.  The  products  of  this  work  would  be  Training  Objectives  and  Lesson  Plans  for 
each  of  the  target  groups.  This  research,  although  not  focused  solely  on  the  commercial 
cockpit,  may  provide  information  on  decision  making  when  the  crews  have  multiple  taking 
(which  is  standard  for  military  aircrews)  and  may  give  guidance  on  how  training  can  be 
developed  for  different  levels  of  an  aviator’s  career,  so  that  he/she  is  given  appropriate 
training  that  will  build  as  professional  skills  are  built. 

C.  Developing  Training  Stratenies 

The  objective  of  this  research  would  be  to  identify  a  spectrum  of  training  strategies  utilizing 
appropriate  media  and  methods  for  the  target  user  groups  (civil,  military  and  general 
aviation).  A  team  of  researchers,  instructional  design  and  media  specialists  was  proposed 
to  develop  a  variety  of  training  materials.  Examples  of  the  proposed  products  included  the 
following: 

1.  Desktop  decisional  aids  (PC-based  with  software  modifications  for  each  target 
group) 

2.  Computer-based  training  devices  (interactive  video  disk) 

3.  Virtual  reality  technology  for  out-of-cockpit  decision  making  exercises 
(Emergency  Medical  Service  helicopters,  tower  cab,  maintenance,  etc.) 

4.  LOFT  scenarios  across  target  groups 

5.  Procedural  changes  to  aid  decision  making  tasks 

6.  Audio-visual  training  aids 

7.  Tailored  manuals  with  task  specific  ADM  training 
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D. 


Evaluating  Training  Effectiveness 


The  objective  in  this  area  is  to  evaluate  the  effectiveness  of  training  strategies  to  meet  the 
training  objectives,  using  appropriate  training  instruments  for  each  of  the  user  groups.  This 
would  require  the  development  of  necessary  experiments,  execution  of  the  experiments, 
data  collection  and  analysis.  Essential  to  this  task  is  the  development  of  an  acceptable 
measure  or  "marker"  of  cognitive  performance  changes  pre-  and  post- ADM  training.  This 
type  of  cognitive  performance  evaluation  tool  requires  further  research  and  development  of 
its  own.  In  particular,  the  use  of  appropriate  cognitive  markers  for  sptecific  task  demands 
identified  during  a  flight  (see  Cognitive  Task  Analysis  discussion  from  Working  Group  A) 
needs  to  be  defined  and  tested. 

7.3  Future  Requirements  and  Recommendations 

Two  of  the  major  areas  addressed  by  this  group  were  definin  -  the  concepts  and  developing 
the  methods  for  examining  ADM,  such  as  developing  a  taxonomy,  etc.  Some  examples 
are:  How  are  decisions  quantified?  Wha'  is  the  process?  If  the  process  is  good,  can  the 
decision  be  poor?  Can  the  process  be  mea.  ired,  or  just  the  outcome?  What  techniques  are 
available  for  that? 

The  major  output  from  Vvorldng  o  B  was  the  specification  of  a  "strawman"  five-year 
plan  to  accomplish  the  work  ou.  wJ  1  e  group’s  estimate  of  the  task  required  to  analyze 
each  problem  area  av  i^uately  ve-  \s  fcilows: 

Problem  Area  Description 

i ..  Identify  ADM  Processes 

B.  Define  ADM  Objectives 
initio/UPT 
Helicopters 

C.  Develop  Training  Strategies 


D.  Evaluate  Training 

Effectiveness 


Tasks 

1 .  Define  Methodology 

2.  Develop  taxonomy 

1.  Transport  aircraft  2.  Ab 

3. 


1.  Military 

2.  Commercial  Operators 

3.  General  Aviation 


1.  Define  Cognitive  Tasks 

2.  Develop  Perf.  Markers 

3.  Perform  Experiments 
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8.0  WORKING  GROUP  C:  ADVANCED  TRAINING  &  EVALUATION 
METHODS 

This  group  was  asked  to  assess  the  need  for  advanced  ADM  training  based  upon  the 
problems  and  issues  identified  during  the  first  day’s  presentations  and  their  combined 
research  and  operational  experience.  The  group  chairman  was  Richard  J.  Adams  of 
Advanced  Aviation  Concepts,  Inc.  (407)  747-3414.  The  co-chair  was  Dr.  Anders  Ericsson 
of  the  University  of  Colorado  (303)  492-1492.  The  subject  matter  experts  and  technical 
support  staff  were: 

Mr.  Roy  G.  Fox 
Mr.  Pete  Hwoschinsky 
Dr.  Judith  Orasanu 
Mr.  Steve  Paul 

8.1  Introduction 

The  deliberations  of  this  working  group  were  based  upon  an  assessment  of  the  deficiencies 
and  problems  of  current  pilot/crew  ADM  training.  Although  the  interface  between  the 
pilots  and  air  traffic  controllers  in  the  decision  making  process  was  included  in  this 
analysis,  the  explicit  consideration  of  the  need  for  controller  decision  making  training,  the 
anticipated  problems  and  research  required  for  ATC  were  outside  the  scope  of  this 
workshop. 

A.  The  purpose  of  the  current  discussions  was  to  identify  and  plan  the  future  work 
required  to  address  these  problems.  The  group  discussion  was  expedited  and  focused  by 
the  use  of  a  three  step  process. 

1.  Identified  ADM  problems  and  needs  in  a  "shopping  list"  format  and  then 
selected  the  top  three  for  more  detailed  analysis. 

2.  Developed  problem  statements  for  the  issues  including  what  needed  to  be 
done  and  which  user  group  was  affected. 

3.  Identified  the  kinds  of  training  products  needed  and  categorized  the  products 
suitable  for  each  user  group. 

B.  This  analysis  included  discussion  of  operational  decision  making  objectives  and 
whether  or  not  they  varied  by  type  of  operation  or  user  group,  i.e..  Commercial,  Military 
or  General  Aviation.  The  discussion  within  each  of  these  groups  addressed  different  types 
of  decisions  required  by  different  aircraft  types  (helicopter  vs.  fixed  wing,  piston  vs. 
turbine,  single  vs.  multi-engine,  etc.).  The  needs  of  both  airplane  and  helicopter  pilots  and 
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crews  were  explicitly  addressed  -  where  they  were  common  and  where  they  were  different. 

Topics  that  could  be  classified  as  the  "structure  of  decision  making  tasks"  were  discussed  at 
length.  These  included:  the  need  to  match  or  tailor  decision  making  training  to  key 
decision  points  in  an  operation  or  mission;  the  desire  to  identify  expert  decision  making 
characteristics  to  match  these  varying  task/ situation  demands;  the  type  of  decision  making 
aiding  that  could  be  developed  to  reduce  the  demands  on  the  crews/pilots;  and,  the  need  to 
alleviate  the  cognitive  dissonance  and  "boomerang"  effects  of  ADM/CRM  training  currently 
observed  by  both  researchers  and  operators. 

The  need  to  match  or  tailor  decision  making  training  to  key  decision  points  in  an  operation 
or  mission  was  considered  to  be  a  worthwhile  future  effort  and  it  was  integrated  into  the 
advanced  training  proposed.  (Note:  Working  Group  A  discussed  this  in  greater  detail  as  a 
Cognitive  Task  Analysis.)  In  addition,  the  need  to  develop  expert  decision  making  training 
to  meet  the  needs  of  critical  tasks  was  recognized  as  a  priority  activity. 

C.  Finally,  the  type  of  future  ADM  training  desired  was  discussed.  The  group  agreed 
that  the  training  needed  to  be  context  based.  Such  training  could  be  subdivided  into 
decision  making  training  with  differing  emphasis:  either  to  anticipate,  or  to  avoid,  or  to 
respond  to  the  task  demands  of  any  normal  or  emergency  situation.  The  consensus  was  that 
the  historical  focus  of  ADM  and  CRM  training  courseware  and  simulator  scenarios  dealt 
with  how  to  respond  to  specific  circumstances  or  problems.  The  group  felt  that  future 
training  should  be  designed  as  follows: 

1 .  Pro-Active 

a.  Anticipating  decision  making  tasks  and  alternatives 

b.  Avoiding  decision  making  situations  (where  possible)  which  involve 
undue  risk 

2.  Reactive:  responding  to  tasks  or  situations  requiring  decisions  after  they 
occur 

Analyzing  decision  making  task  demands  in  this  manner  should  expedite  identification  and 
characterization  of  the  differences  between  expert  and  novice  pilot  decision  strategies  and 
styles.  This  knowledge  can  then  be  used  by  teams  of  senior  captains,  training  officers  and 
researchers  to  develop  expert  training  materials  and  scenarios. 

8.2  Problems  and  Needs 

There  was  significant  discussion  (in  both  content  and  time)  about  the  primary  problems  with 
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ADM  training  and,  in  particular,  with  the  ordering  or  prioritizing  of  those  problems.  Six 
systemic  ADM  problems  were  identified.  Within  these  six,  two  types  of  problems  or  issues 
emerged.  The  first  type  (problems  1,  2,  and  3)  addressed  issues  that  needed  to  be 
addressed  near  term  to  ensure  the  integration  and  success  of  ADM  training.  The  second 
type  (problems  4,  5,  and  6)  addressed  the  future  of  ADM  programs  from  the  perspective  of 
critical  areas  in  need  of  additional  research.  The  specific  problems  in  each  category  were: 

A.  The  need  to  change  attitudes  and  value  acceptance  of  ADM  by  both  pilots  and 
management 

B.  The  need  to  tailor  ADM  training  to  the  non-homogeneous  pilot  community  requiring 
different  solutions. 

C.  The  lack  of  explicit  ADM  training  in  most  operations  (many  Part  121,  most  Part 
135,  majority  Part  91)  and  in  pilot  certification  criteria  of  Part  61. 

D.  How  to  maximize  decision  making  information  and  training  to  the  pilot  community. 
Probably  less  than  5%  of  the  pilot  community  is  being  reached. 

E.  How  to  develop  and  implement  programs  to  train  the  ADM  instructors  and 
evaluators.  Small  general  aviation  fixed  base  operators  involved  in  training  were  of 
particular  concern. 

F.  How  to  evaluate  the  impact  of  ADM  training  experimentally  and  operationally  using: 

1.  controlled  assessments  (before  and  after  simulator  sessions); 

2.  principled  criteria  (i.e.,  theory  grounded);  and 

3.  behavior  assessment  pre-and  post- ADM  or  during  an  evaluation  by  a  check 
airman. 

The  proposed  work  will  support  the  seventh  objective  of  the  FAA’s  National  Plan  for 
Aviation  Human  Factors  which  is:  "to  develop  enhanced  methods  of  training  and  selection 
for  aviation  system  personnel".  [16]  FA  A  policy  now  recognizes  the  need  for  CRM  training 
for  multi-pilot  aircraft  operations  and  there  may  very  well  be  applications  of  the  suggested 
research  for  air  traffic  controller’s  as  well. 

8.3  Advanced  Training  Requirements  and  Recommendations 

The  group  then  discussed  the  requirements  for  sustaining  the  progress  achieved  through 
ADM  thus  far.  This  lead  the  group  to  a  critical  analysis  of  its  future  requirements  to 
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enhance  decision  making  training.  As  often  occurs,  the  use  of  ADM  has  reached  a  plateau 
where  the  training  community  and  the  pilots  are  asking  questions  such  as:  What  do  we  do 
the  next  time  around?  (next  recurrency  training),  and  Can  we  develop  a  second  generation, 
advanced  or  expert  decision  making  curriculum?  The  group’s  response  to  these  questions 
covered  three  primary  areas. 

A.  The  first  area  addressed  the  question:  What  can  we  do  beyond  what  is  being  done 
today  for  decision  making?  The  requirements  that  need  to  be  investigated  in  this  area 
included  those  of  commercial  air  carrier  (Part  121),  commuter  (Part  135),  corporate  (Part 
91)  and  general  aviation.  Each  of  the  last  three  categories  need  to  define  the  varying  needs 
of  both  airplane  and  helicopter  ADM.  The  decision  was  made  to  continue  the  discussion  of 
advanced  training  needs  by  defining  the  components  of  the  training  material  to  be  included, 
and  the  appropriate  training  approaches. 

B.  The  second  area  of  importance  to  the  group  was  centered  on  the  definition  ofthe 
framework  for  problem  solving.  The  group  answered  the  question:  What  do  we  want  to 
train?  A  consensus  answer  was  achieved  that  the  four  areas  of  attitudinal  training, 
knowledge,  procedural  skills  and  process  skills  (including  the  effects  of  time  pressure) 
should  form  the  basis  for  the  framework  of  advanced  ADM.  Although  each  of  these  areas 
have  been  historically  addressed  (both  before  and  after  formal  ADM  training  existed),  a 
more  refined  or  focused  decision  making  approach  was  suggested.  For  example,  the 
following  "advances"  were  proposed  as  potential  fertile  ground  for  the  next  step  in  ADM 
training  using  aircraft  simulators,  interactive  video,  computer  based  training,  classroom 
training,  or  other  advanced  decision  making  training  methods  and  tools. 

1 .  Attitudes  —  Analyze  and  develop  training  required  to  resolve  difficulties 
experienced  with  acceptance  by  some,  resistance  by  others  and  rejection  by  a 
few  (i.e.,  cognitive  dissonance  and  boomerang  effects).  This  will  require  an 
analysis  of  possible  differences  in  expert  and  novice  psychophysiological 
attitudes  including  kinesthetic,  affective  and  cognitive  components,  as  well  as 
exploring  the  role  of  ego  in  decision  making. 

2.  Knowledge  —  Pilot  training  should  include  an  expanded  knowledge  of  the 
importance  of  decision  making,  the  types  of  decisions,  styles  of  decision 
making  and  the  characteristics  that  have  been  recognized  as  "expert"  in 
normal,  emergency  and  novel  situations. 

3.  Procedural  Skills  --  Advanced  ADM  training  should  provide  an 
understanding  of  the  different  types  of  cognitive  skills  required  and  used  to 
satisfy  various  task  demands.  The  difference  between  analytical  reasoning 
and  procedural  knowledge  based  decision  making  should  be  taught,  as  well 
as,  the  need  for  adaptive  decision  making  in  situations  with  novel  cues  or 
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contexts  (e.g,,  UAL  232  catastrophic  failure  accident). 

4.  Process  Skills  --  The  attention,  memory,  recognition  and  recall  skills 
necessary  to  anticipate,  avoid  or  respond  to  inflight  situations  should  be 
incorporated  in  advanced  or  expert  decision  making  training.  Specifically  the 
way  experts  combine  factual  knowledge,  procedural  skills  and  experience  into 
the  problem-solving  process  to  develop  adaptive  plans  needs  to  be 
understood.  This  should  explicitly  address  the  successful  means  of 
responding  within  the  time  frame  required,  that  is,  how  to  attain  the  fast, 
accurate  decision  making  skills  characteristic  of  experts. 

C.  The  third  area  included  in  this  analysis  of  advanced  training  requirements  addressed 
the  question:  How  do  we  want  to  train  decision  making  --  for  both  individual  and  full  crew 
ADM  (training  materials  and  techniques)?  The  response  to  this  query  was  to  develop  a 
comprehensive  list  of  training  methods  and  tools  (listed  in  Table  1)  which  could  be  used  to 
train  and  evaluate  pilot/crew  decisional  skills. 


Table  1  ADVANCED  ADM  TRAINING  TOOLS  AND  TECHNIQUES 


TYPE  OF  TRAINING 

MATERIALS  AND  METHODS 

Individual  Study 

1. 

Manuals  and  Self-tests 

2. 

Other  ADM  publications 

(APS  pubs,  HELIPROPS,  VORTEX) 

Training  Syllabi 

1. 

Student 

2. 

Instructor 

3. 

Evaluator 

Context  Based 

1. 

Classroom  Vignettes 
(structured  hangar  flying) 

2. 

Library  of  Videos 

Experience  Transfer 

1. 

Grand  Rounds  (Mentor)  Concept 

2. 

Peer  Training  (alternate  roles) 

Activity  Based 

1. 

Computer-based  Training 
(Knowledge  based  expert  system) 

2. 

Interactive  video  disk 

Simulator  Scenarios 

I. 

Normal  Procedures 

(based  on  Cognitive  Task  Analysis) 

2. 

Emergency  and  Novel  Situations 
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The  development  of  this  type  of  ADM  training  methods  and  tools  will  supp)ort  the  National 
Plan  for  Aviation  Human  Factors  "Flightdeck  Environment"  requirements  which  include: 
"the  development  of  computer-based  models  of  flightcrew  decision-  making  and  other 
cognitive  processes  to  serve  as  a  basis  for  the  design  of  new  systems". 

The  Advanced  ADM  Recommendations  resulting  from  these  detailed  discussions  (and 
debates)  was  concisely  summarized  by  the  two  primary  recommendations  of  the  group: 

1 .  the  number  one  priority  of  future  efforts  should  be  to  create  the  value  and 
acceptance  of  ADM. 

2.  advanced  training  must  specifically  address  the  diverse  communities 
involved. 

The  areas  needing  attention  in  order  to  effect  the  first  recommendation  include  developing, 
documenting  and  disseminating  the  benefits  of  ADM  for  both  pilots  and  management'.  The 
proposed  Cognitive  Task  Analysis  and  decision  making  taxonomy  will  facilitate  this 
understanding,  without  creating  the  need  for  entirely  new  or  different  training  syllabi. 

Using  these  analytical  techniques,  will  allow  the  training  community  to  express  and 
articulate  both  individual  and  crew  cognitive  behavior  and  performance  in  an  analogous 
manner  to  psychomotor  flying  skills.  Table  2  summarizes  the  recommended  benefit  areas 
that  should  comprise  this  future  work  as  a  minimum. 

Table  2  ADM  VALUE  AND  ACCEPTANCE  AREAS  TO  BE  DOCUMENTED 

COMMUNITY  BENEHTS  TO  BE  QUANTIFIED 

Pilots  Longevity/Livelihood/Quality  of  Life 

Management  Reduced  Accident  Rates/Reduced  Insurance 

Costs/Improved  Public  I  mage/ Improved  Operating 
Benefit/Cost  Ratios 

In  considering  the  specific  work  that  needs  to  be  developed  to  effect  the  second 
recommendation,  we  concentrated  on  the  type  of  training  needed  for  each  of  the  user 
communities.  The  advanced  ADM  training  requirements  were  categorized  according  to  the 
work  generally  needed  by  all  groups,  the  areas  needing  customized  attention,  and  the 
appropriate  tools  that  should  be  considered  for  each  community.  Table  3  presents  the 
consensus  of  the  working  group  for  satisfying  these  needs. 
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Table  3  RECOMMENDED  ADVANCED  ADM  TRAINING  AND  METHODS 
USER 

GROUP  COMMON  NEEDS  CUSTOM  NEEDS  RECOMMENDED  TOOLS 


Part  121 
Part  135 
Part  91 
(incl.  helos) 
Gen.  Aviation 
(incl.  helos) 
Military 


(Reference  Table  1) 


ATTITUDES 

AND 

PROCESS 

KNOWLEDGE 


KNOWLEDGE 

AND 

TAILORED 

SKILLS 


B3,  Cl,  El,  FI  &  F2 
Al,  B2  &  B3  El  &  D2 
Al,  A2,  B2,  Cl  &  D2 

Al,  A2,  Bl,  B2  &  Cl 

Al,  B3,  Cl,  Dl,  E2,  FI  &  F2 


9.0  CONCLUSIONS 

To  return  to  the  focus  of  ADM  and  CRM,  their  similarities  and  differences,  it  should  be 
noted  that  there  was  and  is  ADM  "stand-  alone"  training;  there  was  and  is  CRM  with  ADM 
as  a  part  of  it;  all  current  CRM  does  have  ADM  as  a  component.  But,  ADM  is  the  piece 
of  current  CRM  training  which  now  needs  a  renewed  emphasis,  and  this  will  occur  through 
new,  joint  FAA/NASA/Industry/ Academe  research  and  development  efforts. 

As  with  CRM,  ADM  is  a  growing,  evolutionary  field.  As  with  CRM,  the  early  years 
failed  to  produce  a  well-defined  core  set  of  principles  and  practices.  ADM,  like  CRM,  had 
some  unanswered  questions  and  differing  "camps"  of  thought.  As  with  CRM,  ADM  was, 
and  still  is,  in  need  of  the  development  of  crew  performance/behavior  markers  (or 
evaluation  measures)  to  be  embedded  in  a  new  paradigm  which  focuses  on  the  reality  of 
operational  requirements. 

In  the  past  3  years  CRM  has  begun  to  consensually  develop  and  articulate  accepted  core 
principles  and  practices.  It  has  a  set  of  CRM  performance  markers,  developed  by  Dr. 
Robert  Helmreich  and  used  by  many  air  carriers.  CRM  researchers  and  users  are 
evaluating  at  least  one  new  paradigm  which  does  use  the  mission  as  the  framework  for  the 
analysis  and  evaluation  of  crew  performance  requirements. 

ADM,  by  using  what  has  been  learned  in  CRM  and  by  beginning  with  the  CRM  behavioral/ 
performance  markers  entitled  "Decision  Making"  and  "Leadership",  can  step  off  now  into 
developing  its  own  taxonomy  via  a  Cognitive  Task  Analysis  (CTA).  The  CTA  will  allow 
effective  pilot/crew  behavior  to  be  expressed  in  terms  of  cognitive  and  motor  skills  as  well 
as  attitudes.  The  resultant  would  be  a  new  ADM  paradigm  on  which  to  build  advanced  or 
expert  training  requirements  and  procedures. 

ADM  R&D,  and  the  resultant  insights  and  materials  from  joint  Govemment/Industry/ 
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Research  Community  efforts,  needs  to  provide  future  contributions  in  3  major  aviation 
areas: 

A.  There  will  be  changes  in  operational  procedures.  As  understanding  of  the 
aircrew/airman  decisional  needs  and  processes  grows,  new  light  will  be  shed  on  what 
procedures  are/aren’t  truly  safe  and  efficient--and  why. 

B.  There  will  be  great  changes  in  training:  requirements,  actual  training,  evaluation 
strategies  and  procedures.  There  will  be  emphasis  on  new  paradigms  of  ADM.  These  will 
lead  to  new  training  techniques;  to  recognition  of  the  need  for  new  requirements  for 
providing  crews/pilots  with  the  tools  and  skills  they  need  for  differential  decisions;  to  new 
ADM  training;  and,  to  new  ways  to  evaluate  crew  decisional  needs,  strategies. 

C.  Finally,  there  will  be  new  insights  into  hardware  design,  especially  cockpit  and 
ground  automation,  resulting  from  ADM  R&D.  As  more  is  learned  about  the  decisional 
processes  and  conditions,  this  knowledge  can  be  translated  into  truly  user  friendly 
designs/technology/hardware  that  both  enables  and  actively  assists  in  ADM. 

ADM  materials  and  training,  with  CRM,  hold  the  promise  for  the  major  human  factors 
contribution  to  aviation  safety  in  the  flightcrew  training  and  assessment  areas. 


"LEARN  FROM  THE  MISTAKES  OF  OTHERS;  YOU’LL  NEVER  LIVE  LONG 
ENOUGH  TO  MAKE  THEM  ALL  YOURSELF" . Ralph  Waldo  Emerson 
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APPENDIX  A 


SAMPLE  ADM  WORKSHOP  WORKBOOK 


BACKGROUND 

The  workshop  workbook  was  prepared  and  distributed  for  three  reasons.  First  to  provide 
each  participant  with  a  bound  notebook  to  encourage  recording  of  their  perspectives  of  ADM 
problems,  issues,  requirements,  opinions,  etc.  for  use  throughout  the  workshop.  Second,  to 
provide  a  common  background  knowledge  of  the  expertise  and  interests  of  the  presenters. 
Third,  to  provide  a  basic  foundation  of  the  current  FAA  research  and  publications  in  the 
areas  of  Expert  Pilot  Decision  Making  and  Crew  Resource  Management. 


PURPOSE 

This  appendix  is  provided  in  a  "short"  or  sample  form  of  what  was  provided  the  workshop 
participants.  It  is  intended  to  acquaint  the  reader  with  the  overall  thrust  of  the  workshop  (see 
Section  I  -  Objectives  and  Products),  as  well  as  the  detailed  organization  and  contents  of  the 
agenda  (see  Section  II). 


PRESENTERS 

Section  III  of  this  appendix  is  an  abbreviated  version  of  the  pages  provided  the  participants. 

A  biographical  sketch  of  each  presenter  was  included  (see  Section  7a.  page  7  -  Dr.  Ronald  J. 
Lofaro)  along  with  three  sections  of  mostly  blank  note  taking  space  (see  Section  7b.  pages  8,9  - 
Dr.  Alan  E.  Diehl)  which  gave  the  participants  an  organized  and  lasting  place  to  record: 
Presentation  Notes,  Questions  and  Issues,  and  Future  Required  Activities.  As  shown  on  the 
Agenda  (pages  4,5)  the  discussions  of  the  participants  views  and  ideas  in  these  three  areas 
were  solicited  after  about  every  2  or  3  speakers. 


REFERENCES 

Two  background  papers  were  provided  as  a  bare  minimum  of  common  knowledge  to 
establish  a  starting  point  for  discussion  and  elicit  comments/critiques  from  each  participant's 
perspective  regarding  operational  status  and  research  needs  in  the  areas  of  expert  decision 
making  and  crew  resource  management. 
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AERONAUTICAL  DECISION  MAKING  WORKSHOP 

MAY  6*  and  7*  1992 


A  CONVENING  OF  A  SELECT  GROUP  OF  EXPERTS  AND 
OPERATIONAL  PERSONNEL  ON  AERONAUTICAL  DECISION  MAKING  (ADM) 

FOR  THE  PURPOSE  OF  SHARING  IDEAS 
AND  IDENTHICATION/EXPLORATION  OF  FUTURE  PROJECTS 


THE 

MARRIOTT 
COURTYARD  INN 
DENVER  AIRPORT 


OBJECTIVES: 

1,  TO  DISCUSS  STATE-OF-THE-ART  KNOWLEDGE  AND  ACCOMPLISHMENTS  IN  ADM 
FROM  BOTH  OPERATIONAL  AND  RESEARCH  PERSPECTIVES. 

2,  TO  EXAMINE  THE  DECISION  MAKING  DIFFERENCES  BETWEEN  EXPERT  AND 
NOVICE  PILOTS  ESPEQALLY  THEIR  RESPONSES  TO  NOVEL  CUES,  USE  OF  AUTOMATIC 
INFORMATION  PROCESSING  AND  THEIR  RESISTANCE  TO  ATTITUDINAL  OR 
MANAGEMENT  STYLE  CHANGES. 

3,  TO  FORM  FOCUS  GROUPS  FOR  FOLLOW-ON  JOINT  PROJECTS  IN  THE  NEXT 
GENERATION  OF  ADM  AND  CONSIDER  INTEGRATION  IN  CRM/LOFT  TRAINING. 


PROPOSED  OUTPUTS: 

•  INITIAL  SPECIFICATION  OF  THE  CONTENTS  AND  FORMAT  FOR  THE  NEXT 

GENERATION  OF  ADM  TRAINING. 

•  INITIAL  SPECIFICATION  OF  THE  VALIDATION  METHODS  AND  EVALUATIONS 

NEEDED  TO  INSURE  EFFECTIVENESS  OF  FUTURE  TRAINING. 

•  INITIAL  DEFINITION  OF  AN  ACTION  PLAN  -  INCLUDING  AREAS  OF 

RESPONSIBILITY  FOR  THE  CONTINUED  ENHANCEMENT  OF  ADM  TRAINING. 


34 


AERONAUTICAL  DECISION  MAKING  WORKSHOP 

AGENDA 

Wednesday  TOPICS 

May 

7:30  -  8:00  Continental  Breakfast 
8:00  -  8:15  Welcome  and  Introduction 
8:15  -  8:30  Workshop  Organization  and  Objectives 

Morning  OPERATIONAL  ADM  EXPERIENCES  &  IMPACTS 

8:30  -  9:00  Overview  of  ADM  Effectiveness  Dr.  Alan  E.  Diehl  USAF 

9.00  -  9:30  Decision  Making  Training  Methods  at  Delta  Steve  Paul  Delta 

9:30-10:00  Decision  Making  in  Operational  Systems  Kevin  Smith  Opcon 

10:00  - 10:30  Problems,  Issues  &  Requirements  Development . Group  Discussion 

10:30-10:45  BREAK 

10:45- 11:15  Bootstrapping  Experience:  What  Makes  the 

Expert  Pilot  Decision  Maker  Expert  and  Why?  James  E.  Irving  United 

11:15-11:45  The  Effect  of  Trans-Cockpit  Authority  Gradient 

on  Navy  &  Marine  Aircraft  Mishaps  Dr.  Robert  A.  Alkov  USN 

11:45- 12:15  Problems,  Issues  &  Requirements  Development . Group  Discussion 

12:15  - 1:15  LUNCH  (on  your  own) 

Afternoon  RESEARCH  PERSPECTIVES  FOR  THE  NEXT  GENERATION  ADM 

1:15-1:30 
1:30-2:00 

2:00  -  2:15 
2:15  -  2:45 

2:45-3:00  BREAK 

3:00  -  3:30  A  Cognitive  Model  for  Training  Decision 

Making  in  Aircrews  Dr.  Gary  A.  Klein  Klein  Ass. 

3:30  -  4:00  Elements  of  a  Theory  of  Natural  Decision 

Making  Dr.  John  Bloomfield  Honeywell 

4:00  -  4:30  Shared  Mental  Models  in  Crew  Decision 

Making  Dr.  Judith  Orasanu  NASA 

4:30  -  5:00  Problems,  Issues  &  Requirements  Development . Group  Discussion 


How  Expert  Pilots  Think  Rich  Adams  AAC 

Methodology  for  Studying  and  Training 

Expertise  Dr.  K.  Anders  Ericsson  U  of  CO 

Enhanced  ADM  Training  Alternatives  Rich  Adams  AAC 

Problems,  Issues  &  Requirements  Development . Group  Discussion 


SPEAKER  ORG. 


Dr.  Ronald  J.  Lofaro  FA  A 

Richard  J.  Adams  AAC 
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AERONAUTICAL  DECISION  MAKING  WORKSHOP 

AGENDA 


Thursday  ACTIVITY  MODERATOR 

May  7th 

7:30  -  8:00  Continental  Breakfast 
8:00  -  8:30  Heliprops,  Embedded  ADM  to  Reduce 

Human  Error  Accidents  in  Helicopters  Roy  Fox 
8:30  -  9:00  Working  Group  Organization  &  Activities  Ron  Lofaro 
9:00  -  9:30  Goals  and  Proposed  Outputs  Rich  Adams 

9:30-12:30  Working  Groups  Convene  see  below 

Morning  NEXT  GENERATION  "ADM"  DIRECHONS 

Group  A  Group  B 

FUTURE  CIVIL  AVIATION  FUTURE  MILITARY  AVIATION 
RESEARCH  &  TRAINING  RESEARCH  &  TRAINING 
MODj  Ron  Lofaro  MOD.;  A1  Diehl 

12:00  - 1:00  LUNCH  (on  your  own) 

Afternoon  ADVANCED  RESEARCH  &  TRAINING  PROJECTS  &  PLANS 

1:00-1:30  Working  Group  A  Goals  &  Action  Items  Ron  Lofaro 

1:30  -  2:00  Working  Group  B  Goals  &  Action  Items  Al  Diehl 

2:30  -  3:00  Working  Group  C  Goals  &  Action  Items  Rich  Adams 

3:00-3:15  BREAK 

3:15  -  4:00  Proposed  Projects,  Products  &  Plans  Ron  Lofaro  & 

Rich  Adams 


Group  C 

ADVANCED  TRAINING  & 
EVALUATION  METHODS 
MOD.:  Rich  Adams 
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PRESENTERS 


Dr.  Ronald  J.  Lofaro 


Aviation  Psychologist/Project  Manager 
Federal  Aviation  Administration  (FAA) 


Dr.  Lofaro  has  been  with  the  FAA  at  their  national  headquarters  (HQ)  since  1989  as  an 
aviation  Psychologist /Project  Manager  for  Human  Factors  research  and  development  in 
aviation  safety.  He  is  working  on  such  issues  as  evaluation  techniques  for  cockpit  resource 
management  (CRM)  and  line  oriented  simulation/line  oriented  flight  training  (LOS/LOFT); 
the  integration  and  assessment  of  CRM  and  Flight  Control  Skills,  the  identification  and 
training  of  Expert  Pilot  Decision  Making  (EDM);  air  traffic  controller  selection/screening  and 
circadian  dysrhythmia /desynchronosis  issues  in  both  long  and  short-haul  flying. 

Prior  to  coming  to  the  FAA,  Dr.  Lofaro  was  with  the  Army  Research  Institute's  (ARI) 
Aviation  R&D  Unit  at  the  Army's  Aviation  Center  in  Alabama.  He  focused  on  various 
aspects  of  aviation  safety  and  training  including:  aviator  candidate  classification  and 
Selection;  air-to-air  combat  human  factors;  aviation  accident  investigation/ analysis. 

Previous  to  ARI,  he  was  involved  in  human  resources  with  the  FAA's  Eastern  Region,  as  a 
personnel  Psychologist  for  the  Army's  Training  and  Doctrine  Command,  involved  in  test 
development  and  evaluation  and  served  as  a  clinical  psychologist  at  Walson  Army  Hospital. 
Dr.  Lofaro  spent  twelve  years  as  a  university  professor  after  serving  in  the  US  Air  Force  from 
1960  to  1965. 

Dr.  Lofaro  earned  his  PhD.  from  the  New  York  University.  He  is  widely  published  in  aviation 
psychology  including  the  Human  Factors  Society  Journal  and  Bulletin  and  the  International 
Journal  of  Aviation  Psychology.  He  is  a  member  of  the  SAE  Human  Behavior  Technology 
Committee. 
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Dr.  Alan  E.  Diehl 


Human  Performance  Technical  Advisor 
U.S.  Air  Force  Safety  Agency 


Dr.  Alcin  E.  Diehl  has  been  the  Human  Performance  Technical  Advisor  for  the  US  Air  Force  Safety  Agency  since 
1987.  He  holds  acadenuc  degrees  in  psychology,  management  and  engineer  and  holds  Airline  Transport  Pilot  and 
Flight  Instructor  ratings.  He  has  several  imp>ortant  accomplishments  and  distinctions  resulting  from  his  twenty 
years  of  professional  exp>erience  with  US  aircraft  industry  and  government  agencies. 

At  LTV  Electrosystems,  Dr.  Diehl  assisted  in  the  design  of  Project  "Redflame,"  one  of  the  first  lasers  to  be  used  in 
combat.  With  Cessna  Aircraft ,  his  respx>nsibilities  included  flight  deck  design  on  the  Qtation  business  jet.  He 
later  won  the  esteemed  Collier  Trophy  for  the  Qtation's  outstanding  safety  record. 

As  an  Aviation  Research  Psychologist  for  the  Naval  Training  Systems  Center  ,  Dr.  Diehl  developed  iimovative 
methods  for  evaluating  simiilator  training  programs.  Joining  the  NTSB  HQ  staff  in  1977,  he  was  the  first  Human 
Performance  Scientist/ Air  Safety  Investigator,  helping  to  expand  the  scope  and  effectiveness  of  investigations 
and  to  better  document  the  underlying  causes  of  human  error.  He  drafted  the  first  recommendation  calling  for 
opTerational  implementation  of  Cockpit  Resource  Management  (CRM). 

He  became  the  Program  Scientist  for  Human  Performance  at  FA  A  HQ  in  1980,  directing  research  into  pilot 
judgment  training,  monitoring  ongoing  work  in  CRM  and  initiating  development  of  several  training  manuals  and 
audio-visual  programs  related  to  cockpit  management  and  aeronautical  decision  making  (ADM).  Dr.  Dehl 
received  two  sp>ecial  achievement  awards  for  his  work  with  the  FAA. 

Since  transferring  to  the  USAF,  he  continues  to  serve  as  a  consultant  to  mishap  investigation  boards.  He  has 
help)ed  various  commands  select  and/or  develop  cockpit  management  pnt>grams.  He  has  also  served  as  a  consultant 
to  other  important  safety  efforts  such  as  ICAO,  Flight  Safety  and  Hiunan  Factors  Study  Group,  and  the  DOD  Air 
Carrier  Analysis  System. 


Presentation:  "Overview  of  ADM  Effectiveness" 


Presentation  Notes: 
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Questions/Issues: 


Future  Required  Activities: 


/ 


Capt  Steve  Paul  Senior  Facilitator,  Program  Developer 

Delta  Airlines 

Captain  Steve  Paul  is  currently  a  senior  facilitator  and  program  develojjer  for  Delta  Airlines. 
His  responsibilities  include  developing  programs  for  check  airmen,  training  and  initial 
Captain  upgrade  training.  This  includes  assisting  with  the  development  for  recurrent 
training  and  facilitator  training.  Facilitation  activities  include  check  airmen,  recurrent  and 
various  co-facilitated  flight  attendant  training. 

Captain  Paul  graduated  from  the  University  of  Colorado  in  1966  with  a  major  in  Psychology. 
He  served  in  the  US  Navy  as  a  pilot  as  a  T-39  NATOPS  instructor  and  check  airman.  As  a 
pilot  for  Delta,  he  has  flown  the  727  as  both  Second  and  First  Officer  (SO  &  FO  respectively); 
the  DC-8  as  SO;  IXZ-9  as  FO;  the  LlOll  as  FO  and  the  737  as  Captain.  He  was  a  CRM  facilitator 
from  1989  to  1990. 


Presentation:  Decision  Making  Training  Methods  at  Delta 

Presentation  Notes: 

Questions/Issues: 

Future  Required  Activities: 


Kevin  Smith  Captain 

United  Air  Lines 

Kevin  Smith  is  currently  a  captain  on  the  737/300  and  a  First  Officer  on  the  747/400.  He 
served  two  years  at  the  United  Air  Lines  Training  Center  working  on  CRM  and  AQP  and  was 
a  member  of  the  Air  Transport  Association  (AT  A)  Committee  on  CRM /AQP/ LOS.  He  has 
coordinated  and  worked  with  Dr.  Helmreich  and  the  FAA  Flight  Standards  and  R&  D. 

Captain  Smith  served  as  a  Navy  pilot  flying  the  FlllB  Tiger  and  has  extensive  experience 
with  both  military  and  civil  operations  analysis. 


Presentation:  Decision  Making  In  Operational  Systems 


Presentation  Notes: 
Questions/Issues: 

Future  Required  Activities: 
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James  E.  Irving 


Captain 
United  Air  Lines 


Captain  Jim  Irving  flew  with  the  US  Navy  for  five  years  in  helicopters  and  multi-engine 
patrol  aircraft  as  well  as  jet  fighters.  As  in  instructor  pilot  for  Lockheed  Commercial  Aircraft 
on  the  Lion,  he  developed  classroom  training  and  performed  simulator  and  aircraft  training 
for  airlines  all  over  the  world. 

Capt.  Irving  was  director  of  Learjet  training  at  Martin  Aviation  in  Santa  Ana,  Califonua,  one 
of  the  largest  Learjet  training  facilities  in  the  United  States.  During  this  period  he  also 
developed  and  managed  the  ground  school. 

Jim  has  been  flying  with  United  Air  Lines  for  eighteen  (18)  years  and  is  currently  a  Captain 
with  United  Airlines  flying  a  Boeing  737/300.  His  experience  includes  pilotage  in  captain,  first 
and  second  officer  positions  in  half  a  dozen  aircraft  tjqjes  he  has  been  training  check  airman. 

Capt.  Irving  is  keenly  interested  in  the  problems  of  training  commercial  airline  pilots,  in  the 
particular  problems  presented  by  advanced  automated  aircraft,  and  is  providing  subject 
matter  "expertise"  to  a  team  at  the  University  of  Colorado  studying  part-task  training. 


Presentation:  Bootstrapping  Experience:  What  Makes  the  Expert  Pilot 
Decision  Maker  Expert  and  Why 


Presentation  Notes: 
Questions/Issues: 

Future  Required  Activities: 


Dr.  Robert  Alkov  Research  Psychologist 

Naval  Safety  Center 

Robert  A.  Alkov  received  a  PhD.  in  Experimental  Psychology  from  Florida  State  University 
in  1965.  He  worked  as  an  aviation  psychologist  for  George  Washington  University 
(HumRRO)  under  contract  to  the  US  Army  for  aviation  training  research  at  Ft.  Rucker, 
Alabama  in  the  mid  1960s.  he  has  been  employed  at  the  Naval  Safety  Center  as  a  research 
psychologist  for  the  past  25  years. 

Bob  is  a  retired  Naval  Reserve  Aviator,  a  Fellow  of  the  Aerospace  Medical  Association  and 
past  president  of  both  the  Association  of  Aviation  Psychologists,  and  the  Tidewater  chapter  of 
the  Human  Factors  Society.  He  teaches  graduate  courses  for  Embry-Riddle  aeronautical 
University. 
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Since  1987,  Dr.  Alkov  has  been  working  to  introduce  aircrew  coordination  training  into  US 
Navy  aviation  training  squadrons  and  units,  he  has  been  employed  at  the  Naval  Safety 
Center  as  a  research  psychologist  for  the  past  25  years. 

Bob  is  a  retired  Naval  Reserve  Aviator,  a  Fellow  of  the  Aerospace  Medical  Association  and 
past  president  of  both  the  Association  of  Aviation  Psychologists,  and  the  Tidewater  chapter  of 
the  Human  Factors  Society.  He  teaches  graduate  courses  for  Embry-Riddle  aeronautical 
University. 

Since  1987,  Dr.  Alkov  has  been  working  to  introduce  aircrew  coordination  training  into  US 
Navy  aviation  training  squadrons  and  units. 

Presentation:  The  Effect  of  Trans-Cockpit  authority  Gradient 
on  Navy  &  Marine  Aircraft  Mishaps 


Presentation  Notes: 

Questions/Issues: 

Future  Required  Activities: 

Richard  J.  Adams  Vice  President 

Advanced  Aviation  Concepts,  Inc 

Mr.  Adams  received  his  B.S.  in  Aeronautical  and  Astronautical  Engineering  from  the 
University  of  Illinois  in  1965  and  his  M.S.  in  Mechanical  Engineering  from  the  University  of 
Florida  in  1968.  Mr.  Adams  is  a  registered  Professional  Engineer  in  Mechanical  Engineering 
in  the  State  of  Florida  and  a  private  pilot.  Mr.  Adams  has  been  involved  in  civil  aviation 
research  and  development  for  over  27  years.  He  has  authored  over  sixty  (80)  technical 
reports,  articles  and  papers  in  the  fields  of  pilot  training,  pilot  workload,  accident  data 
analysis,  avionics,  area  navigation,  aircraft  energy  conservation,  air  traffic  management, 
propulsion  and  controls.  From  1974  to  1984  Mr.  Adams  specialized  in  the  technical  and 
operational  problems  of  the  dvil  helicopter  operator.  Since  1985,  he  has  supported  the  FAA's 
Aeronautical  Decision  Making  research  in  the  development  of  decision  making  training 
manuals  for  private,  commercial,  instrument  and  instructor  pilots,  emergency  medical 
service  operators  and  hospital  administrators.  He  has  investigated  accident  trends  to 
determine  where  training  can  be  enhanced  and  recommend  ways  of  changing  pilot  attitudes 
to  improve  safety. 
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Presentation:  How  Expert  Pilots  Think 


Presentation  Notes: 
Questions/Issues: 

Future  Required  Activities: 


Richard  J.  Adams  (Continued) 

Presentation:  Enhanced  ADM  Training  Alternatives 


Presentation  Notes: 
Questions/Issues: 

Future  Required  Activities: 


Dr.  K.  Anders  Ericsson  Associate  Professor,  Psychology 

University  of  Colorado,  Boulder 

K.  Anders  Ericsson  received  his  Ph.D.  in  Cognitive  Psychology  in  1976  from  the  University  of 
Stockholm  (Sweden).  For  two  years  he  headed  a  group  in  Human  Factors  at  the  Institute  of 
Aviation  Medicine  in  Linkoping,  Sweden,  where  he  was  involved  in  task  analysis  and 
cockpit  design  for  the  next  generation  of  light-attack  airplanes  for  the  Swedish  Airforce. 
During  1977  to  1980  he  was  a  post-doctoral  fellow  at  Camegi  e-Mel  Ion  University  in 
Pittsburgh,  working  with  Professor  Herbert  Simon  (Nobel  Prize  1978). 

He  has  served  as  Assistant  Professor  of  Psychology  at  the  University  of  Colorado  and  the  Max- 
Planck  Institute  for  Human  Development  and  Education,  West  Berlin.  He  recently  accepted 
the  position  of  Coirradi  Eminent  Scholar  Chair  of  Psychology  in  January  1993  at  the  Florida 
State  University  to  develop  research  on  the  study  of  expert  performance.  His  current  research 
interest  concerns  the  study  of  expert  performance-its  structure  and  acquisition.  Dr.  Ericsson  is 
widely  published  including  books  with:  Herbert  Simon,  "Protocol  analysis:  Verbal  Reports  as 
Data;"  and  Jacqui  Smith,  'Toward  A  General  Theory  of  Expertise:  Prospects  and  Limits." 


Presentation:  Methodology  for  Studying  and  Training  Expertise 


Presentation  Notes: 
Questions/Issues: 

Future  Required  Activities 
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Dr.  Gary  A.  Klein 


Chairman,  Chief  Scientist 
Klein  Associates  Inc 


Dr.  Klein  received  his  Ph.D.  in  experimental  psychology  from  the  University  of  Pittsburgh  in 
1969  and  served  as  an  Assistant  Professor  of  Psychology  at  Oakland  University  until  1973.  As 
a  research  psychologist  at  the  Air  Force  Human  Resources  Laboratory  from  1974  to  1978,  he 
was  a  member  of  the  team  that  investigated  the  use  of  functional  requirements  to  design 
aircrew  simulators. 

In  1978,  Dr.  Klein  formed  Klein  Associates  conducting  studies  of  naturalistic  decision  making 
at  both  the  levels  of  individual  and  team  decision  making.  He  has  developed  methods  of 
cognitive  task  analysis  for  use  in  conducting  research  and  in  developing  training  materials. 
Additional  applications  of  this  work  include  the  development  of  storyboards  for  man- 
machine  interfaces  and  knowledge  elicitation  for  market  research  of  consumer  decision 
making.  He  has  developed  the  Recognition-Primed  Decision  Model  which  describes  how 
people  make  rapid  decisions  in  a  domain  with  which  they  have  experience. 


Presentation:  A  Cognitive  Model  for  Training 
Decision  Making  in  Aircrews 


Presentation  Notes: 
Questions/Issues: 

Future  Required  Activities: 


Dr.  John  Bloomfield  Honeywell  Systems  and  Research  Center 

Dr.  John  Bloomfield  was  educated  in  England,  obtaining  a  B.A.,  in  Psychology  from  the 
University  of  Hull  and  a  PhD  in  Psychology  from  the  University  of  Nottingham.  In  1972,  he 
moved  to  the  United  States  to  carry  out  post-doctoral  research  at  the  Ohio  State  University 
Research  Foundation,  and  then  joined  Honeywell  two  years  later.  Dr.  Bloomfield  is 
interested  in  natural  decision  making  and  knowledge  acquisition  techniques,  he  developed  a 
flight  decision  taxonomy,  while  working  on  a  Wright-Patterson  AFB  program,  then  used  this 
as  the  basis  for  a  more-generally  applicable  Decision  Categorization  Framework.  Also,  he  has 
developed  an  advanced  knowledge  acquisition  technique,  Knowledge-Sensitive  Task 
manipulation,  and  demonstrated  its  use  of  acquiring  specific  detailed,  tacit  knowledge  from 
commercial  airline  pilot  training  instructors  flying  a  motion-based  flight  simulator. 
Currently,  Dr.  Bloomfield  is  Principal  Investigator  on  an  FAA's  program,  the  objective  of 
which  is  to  investigate  Controlled  Flight  Into  Terrain  (CFIT)  air  accidents.  In  addition  to  his 
work  at  Honeywell,  Dr.  Bloomfield  is  an  Adjunct  Professor  at  the  University  of  St  Thomas 
and  at  Hamline  University. 
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Presentation:  Elements  of  a  Theoiy  of  Natural  Decision  Making 


Presentation  Notes: 
Questions/Issues: 

Future  Required  Activities: 


Dr.  Judith  Orasanu  Flight  Human  Factors  Branch 

NASA-Ames  Research  Center 

Dr.  Orasanu  received  her  PhD  in  Experimental  Psychology  from  Adelphi  University  in  1975 
and  was  awarded  a  Postdoctoral  Fellowship  at  Rockefeller  University.  As  a  Research 
Associate  at  Rockefeller  she  focused  on  cognitive  processes  in  comprehension.  Pr.  Orasanu 
joined  the  Crew  Factors  Group  at  NASA-Ames  Research  Center  in  1991  where  she  is 
conducting  research  on  crew  problem  solving  and  decision  making.  She  went  to  NASA  after 
six  years  with  the  US  Army  Research  Institute's  Basic  Research  Office.  In  that  position  she 
developed  a  program  on  Planning,  Problem  Solving  and  Decision  Making  that  examined 
reasoning  in  everyday  settings.  Her  involvement  with  that  program  laid  the  foundation  for 
research  on  crew  communication  and  problem  solving,  initiated  when  she  was  a  visiting 
Fellow  at  Princeton  University  (1989-1990).  Prior  to  joining  ARI,  Dr.  Orasanu  was  a  Senior 
Research  Associate  at  the  National  Institute  of  Education  managing  research  on  literacy  and 
bilingualism  from  1979  to  1983. 

Dr.  Orasanu  is  widely  published  in  the  areas  of  CRM,  stress  and  performance  and  decision 
making.  She  edits  a  series  on  Cognition  and  Literacy  for  Ablex  Publishers. 


Presentation:  Shared  Mental  Models  in  Crew  Decision  Making 


Presentation  Notes: 
Questions/Issues: 

Future  Required  Activities: 


Roy  G.  Fox  Chief  Safety  Engineer, 

B^l  Helicopter  Textron 

Mr.  Fox  started  the  Systems  Safety  Engineering  Group  at  Bell  and  still  directs  that  effort.  In 
addition  to  Safety  Engineering,  he  is  deeply  involved  in  crash  survival  of  helicopters.  This 
includes  military  and  civil  helicopters.  Mr.  Fox  has  chaired  the  helicopter  industry  AIA 
(Aerospace  Industries  Association)  Crashworthiness  Project  Group,  member  of  General 
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Aviation  Safety  Panel  for  seats  and  post  crash  fire  protection,  and  member  of  SAE  (Society  of 
Automotive  Engineers)  committees  on  aircraft  seats  and  occupant  restraints,  he  also  lectures 
on  crash  survival  and  human  performance  at  the  Bell  pilot  training  s-'hool  and  the  FAA 
Helicopter  Safety  and  Accident  Investigation  Course.  He  also  provides  HELIPROPS  safety 
briefings  at  regional  safety  seminars. 

In  1989,  Mr.  Fox  received  the  AHS  (American  Helicopter  Society)  Harry  T.  Jensen  award  for 
significant  improvements  in  reliability,  maintainability,  or  safety  due  to  his  crash  survival 
efforts  that  are  becoming  mandatory  FAA  rules.  In  1990,  Mr.  Fox  received  the  FSF  (Flight 
Safety  Foundation)  Business  Aviation  Meritorious  Award  for  his  crash  survival  efforts  and 
teaching  efforts  that  are  resulting  in  improved  safety  in  business  aviation. 


Presentation:  "Heliprops,  Embedded  ADM  to  Reduce  Human  Error  Accidents  in  Helicopters" 


Presentation  Notes: 
Questions/Issues: 

Future  Required  Activities: 


Working  Group  Goals  and  Action  Items 
Working  Group  A-Notes/Issues 


Working  Group  Goals  and  Action  Items 
Working  Group  B-Notes/Issues 


Working  Group  Goals  and  Action  Items 
Working  Group  C-Notes/Issues 


Proposed  Projects,  Products  &  Plans 
Notes/Issues 
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TRAINING  CONSIDERATIONS  FOR 
EXPERT  PILOT  DECISION  MAKING 

Decision  making  training  has  made  a  significant  impact  on  safety  by  reducing  the  human  performance 
error  related  accidents  in  both  civil  and  military  aviation.  Accident  rate  reductions  of  about  50%  can  be 
seen  when  comparing  pilot  groups  with  and  without  decision  making  training.  Yet,  these  large  safety 
improvements  have  not  reached  the  entire  pilot  community.  In  particular,  there  is  a  lack  of  acceptance 
of  the  linear  decision  making  model  by  the  more  experienced  pilots,  a  resistance  to  change  in 
interpersonal  skills  in  the  multicrew  environment  and  even  a  "boomerang"  effect  where  the  attitude 
toward  the  use  of  all  crew  resources  deteriorates  after  training. 

Extensive  research  and  empirical  testing  in  Aeronautical  Decision  Making  (ADM)  produced  a  series  of 
ten  Federal  Aviation  Administration  manuals  and  refx>rts  on  ADM  (1986-1988).  Although  it  is 
admittedly  difficult  to  accurately  assess  the  impact  of  all  of  the  manuals  throughout  aviation, 
significant  reductions  in  human  performance  error  (HPE)  accidents  have  been  documented  in  specific 
areas: 


•  A  36%  reduction  in  all  HPE  accidents  for  the  worldwide  B206  fleet  (Fox  1991). 

•  A  72%  reduction  in  weather  related  HPE  accidents  for  the  worldwide  B206  fleet. 

•  A  48%  reduction  in  all  HPE  accidents  for  the  U.  S.  B2()6  fleet. 

•  A  54%  reduction  in  HPE  accidents  for  the  largest  U.  S.  civil  helicopter  operator 

(more  than  two  million  takeoffs  and  landings  annually) 

•  A  51%  reduction  in  accidents/l()0,000  hours  for  USAF  MAC  crews  (Diehl  1991). 

•  A  20%  reduction  in  USN  helicopter  air  crew  mishap  rate  and  an  81%  reduction 

by  A6  and  EA6  airplane  pilots  (Alkov  1991) 

These  findings  generated  a  request  from  many  in  the  aviation  community  for  advanced  decision  making 
and  crew  resource  management  material.  They  also  have  led  the  National  Transportation  Safety 
Board  to  recommend  that  the  FAA  pursue  the  implementation  of  ADM  more  vigorously  following  a 
fatal  1991  accident  involving  an  airplane  and  a  helicopter  (NTSB  1991)  However,  as  dramatic  as  the 
examples  of  improvements  were,  a  more  detailed  examination  of  the  accident  rate  reduction  data 
disclosed  that  the  major  positive  impact  has  been  on  the  less  experienced  pilots.  This  finding  led  to 
two  questions:  Can  we  achieve  the  same  impact  in  human  error  reduction  with  more  experienced  pilots? 
And,  how  can  this  be  done? 

The  current  research  effort  attempts  to  respond  to  these  questions,  questions  which  industry  has  also 
asked.  The  research  is  based  upon  parallel  events  occurring  in  the  air  carrier  industry  during  the  1983- 
1989  timeframe.  During  this  period,  there  were  several  extraordinary  accidents  involving  multiple 
engine  failures,  explosive  decompressions  caused  by  structural  failures,  fuel  starvation  and  in-flight 
fires.  In  each  of  these  accidents,  experienced  pilots  quickly  responded  to  emergencies  for  which  there 
was  no  handbook  procedure  or  previous  training.  They  assessed  the  situation  and  integrated 
airmanship  skills,  trained  procedures  and  aeronautical  knowledge  into  a  quick,  effective  decision 
making  process.  Such  dynamic  cognitive  behavior  was  in  direct  contrast  to  the  more  basic  ADM  training 
which  stressed  a  linear,  measured  approach  to  situation  analysis. 

EXPERT  COGNITIVE  PROCESSES 

Research  in  the  last  twenty  years  has  revealed  that  superior  performance  is  most  often  the  result  of  the 
interaction  between  accumulated  skill  and  experience.  The  primary  differences  between  a  beginner  and 
an  expert  can  be  attributed  to  acquired  knowledge  and  problem  solving  skills:  what  we  call  expertise 
and  which  is  demonstrated  through  performance.  Expert  performance  can  be  defined  as  the  selection  of 
an  appropriate  response  to  situations  or  problems  in  a  wide  variety  of  domains.  These  include  selecting 
the  test  move  in  a  chess  game,  correctly  diagnosing  a  medical  problem,  or  using  the  propjer  emergency 
procedure  in  aviation.  The  relevant  research  on  exf>ert  performance  has  focused  on  the  basic 
understanding  of  knowing  how  to  do  something  well  rather  than  knowing  what  the  underlying 
mechanism  was  for  superior  performance.  Complex  problem  solving  research  assumed  that  the 
integration  of  the  basic  human  information  processing  skills  was  required.  This  included  the  processes 
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of  perception,  memory,  attention,  and  reasoning.  This  research  had  real-world  importance  since 
performance/expertise  obviously  depended  on  learning  how  to  do  something  well. 

All  human  cognition  is  task  dependent  and  purposeful  (goal  oriented).  Humans  use  their  knowledge, 
cognitive  processing  skills  and  the  cues  or  stimuli  of  a  situation  or  task  to  develop  problem  solving 
approaches.  To  accomplish  this,  two  types  of  knowledge  are  used.  These  are  declarative  knowledge 
and  procedural  knowledge.  Declarative  knowledge  consists  of  knowledge  that  can  be  verbalized,  some 
say  knowledge  about  "facts  and  things".  Procedural  knowledge  is  knowledge  about  actions  or  how  to 
perform  various  cognitive  activities.  These  ordinarily  cannot  be  completely  or  adequately  verbalized, 
for  example,  how  to  ride  a  bike.  Procedural  knowledge  is  the  basis  for  development  of  specific  steps 
(also  call^  production  rules)  to  be  used  in  problem  solving  situations.  The  study  of  procedural  learning 
became  a  crucial  area  to  be  understood.  The  current  (general)  theory  of  acquiring  expertise  includes  the 
following  three  stages: 

1 .  Novice's  solve  problems  by  weak,  domain  general,  heuristic  methods  (often  working  backwards  from 
the  goal). 

2.  Successful  solutions  (when  repeated  frequently)  lead  to  the  development  of  domain  specific 
procedures  or  production  rules.  TT^ese  rules  specify  actions  that  will  achieve  goals  under  particular 
conditions.  Production  rules  form  the  beginnings  of  expertise. 

3.  As  these  rules  are  used  more  and  more  often,  and  applied  to  many  situations  in  a  domain,  they  result 
in  fairly  automatic  generation  of  specialized  productions  which  often  use  forward  inferencing  to 
progress  from  the  initial  problem  state  toward  a  solution  or  goal.  Thus,  relative  to  the  novice,  the 
expert  is  able  to  reach  the  correct  solution  more  quickly  and  efficiently. 

The  status  of  these  theories  of  expertise  are  presented  in  two  references  which  provide  24  "Summary 
Propositions"  pertinent  aviation.  Thoughts  on  Expertise  (Glaser  1987)  and  On  the  Nature  of  Expertis 
(Glaser  and  Chi  1988)  provide  the  following  relevant  findings; 

1.  Expert  performance  is  characterized  by  rapid  access  to  a  well  organized  body  of  conceptual  and 
procedural  knowledge.  High  levels  of  competence  result  from  the  interaction  between  knowledge 
structure  and  processing  abilities. 

2.  The  organization  of  knowledge  used  by  experts  can  be  thought  of  as  schemata  or  a  modifiable 
information  structure  based  upon  knowledge  that  is  experienced.  Schema  theory  assumes  there  are 
schemata  for  recurrent  situations  that  expedite  decisions  in  certain  situations. 

3.  Expertise  is  domain  specific.  Within  a  domain,  experts  develop  the  ability  to  perceive  large 
meaningful  patterns.  Furthermore,  the  expert’s  pattern  recognition  occurs  so  rapidly  that  it  appears  to 
take  on  the  character  of  insight  or  intuition. 

4.  Expert  knowledge  is  highly  procedural  and  goal  oriented.  Individuals  with  extensive  domain 
knowledge  are  much  better  at  relating  events  in  cause-and-effect  sequences  that  relate  to  the  goals  and 
subgoals  of  a  problem  solution. 

5.  The  capability  of  experts  to  fast-access  their  knowledge  facilitates  problem  perception  in  a  way 
that  leads  to  the  reduction  of  the  role  of  memory  search  and  general  processing.  The  outstanding 
performance  of  experts  is  derived  from  how  their  knowledge  is  structured  to  accomplish:  Retrieval, 
Pattern  Recognition,  and  Inference. 

6.  Generalized  thinking  and  problem  solving  skills  nriay  develop  in  individuals  who  acquire  expertise 
in  several  domains  (e.  g.,  aeronautics,  airplane  systems,  air  traffic  control  procedures,  emergency 
procedures,  etc).  Continuous  development  of  expertise  in  a  field  is  based  upon  novel  conditions  that 
extend  competence  to  novel  situations. 
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7.  Experts  develop  specialized  schemata  that  match  goals  to  demands  of  the  problem.  Although  both 
novices  and  experts  can  display  good  use  of  general  problem  solving  process,  experts  use  them  primarily 
in  unfamiliar  situations. 

8.  The  development  of  expertise  is  influenced  by  task  demands  encountered  in  the  course  of  experience. 
In  some  domains,  experts  develop  the  capability  for  "opportunistic  plaiuting"  which  enables  them  to 
revise  problem  representations  and  to  access  multiple  possible  interpretations  of  a  situation. 

9.  Experts  also  develop  types  of  metacognition  or  self-regulatory  capabilities  that  are  not  present  in 
less  experienced  decision  makers.  Experts'  skilled  self-regulatory  processes  free  their  working  memory 
for  higher  level  conscious  processing.  These  include;  planning  ahead,  efficiently  monitoring  one’s  time 
and  attentional  resources,  and  monitoring  and  editing  one's  efforts  to  solve  a  problem. 

10.  An  important  point  of  distinction  is  that  there  are  both  routine  and  adaptive  experts.  Adaptive 
experts  possess  the  ability  to  creatively  respond  to  novel  situations  and  develop  an  appropriate 
response  with  some  reasonable  chance  for  a  successful  outcome. 

This  distinction  between  routine  and  adaptive  experts  leads  to  the  threshold  of  the  next  generation  of 
expertise  theory  which  relies  on  a  cognitive  psychology  perspective.  The  current  FAA  sponsored  R&D 
will  examine  applications  to  real  world  problems  and  the  focus  will  be  on  how  aviators  respond  to 
untrainable  emergencies.  A  broad  distinction  between  two  classes  of  expertise  is  suggested  in  Sloboda, 
(1991).  His  definition  is  that  expert  performance  involves  "the  reliable  attainment  of  specific  goals 
within  a  sp)ecific  domain".  An  extended  definition  is  that  "an  exp>ert  is  someone  who  can  make  an 
appropriate  response  to  a  situation  which  contains  a  degree  of  unpredictability". 

In  general,  an  expjert  will  succeed  in  identifying  and  adapting  to  the  inherent  constraints  of  the  task.  If 
the  task  can  be  done  most  efficiently  by  forward  search,  the  expjert  will  search  forward;  if  backward 
search  is  better,  the  expjert  searches  backward.  If  certain  patterns  of  cues  are  crucial  to  pjerforming  the 
task  well,  the  expjert  will  likely  pjerceive  and  remember  them;  if  patterns  are  not  so  important,  the 
expjert  will  not  selectively  process  them. 

EXPERTISE  AND  TRAINING  OR  PRACTICE 

Initially,  expjert  pjerformance  and  expjertise  involves  the  development  of  encoding  processes  which 
allow  the  situation  to  be  fully  represented  and  integrated  cognitively.  In  this  way,  relevant  actions  can 
be  retrieved  from  memory.  The  internal  representation  of  external  situations  is  also  critical  to  planning 
and  evaluation  of  possible  courses  of  action  as  well  as  a  means  to  represent  a  dynamically  changing 
environment  for  the  purpjoses  of  anticipjation  and  prediction. 

It  seems  that  acquisition  of  expjertise  can  be  increased  for  most,  if  not  all,  relevant  aspjects  of 
pjerformance.  Table  2  shows  a  phase,  or  stage,  view  of  differing  levels  of  expjertise. 


Table  2  PHASES  AND  CATEGORIES  OF  EXPERTISE 


PHASES  OF  EXPERTISE  CATEGORY  OF  EXPERT 

Beginning  Phase  (Acquisition  of  declarative  knowledge  Beginner,  Student,  or 

and  domain  general  problem  solving  skills)  Novice 

About  1-2  years  of  active  expjeriencc  and  training  Intermediate 

Many  years  of  active  experience  and  training  Routine  Expjert 

(Full  time  -  40-80  hours  pjer  week)  (or  "Journeyman") 


More  than  10  years  of  full  time  expjcrience 
and  training  _ 


Master  or  Adaptive  Expjert 
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At  this  time,  knowledge  about  how  experts  attain  the  base  for  their  expert  performance  is  relatively 
limited.  Generally  speaking,  the  current  view  is  that  the  novice  should  have  acquired  all  basic 
knowledge  in  less  than  one  year.  Continuing  beyond  this  basic  knowledge  leads  to  the  acquisition  of 
problem  solving  skills  where  the  knowledge  is  organized  to  effectively  produce  efficient  performance. 
That  is,  there  is  an  acquisition  of  the  procedural  knowledge  of  complex  patterns  occurring  in  specific 
situations.  At  this  Intermediate  level,  differences  in  expertise  appear  to  be  related  to  the  cued  recall 
ability  and  the  number  and  complexity  of  those  patterns  available  for  use.  Finally,  in  both  the  routine 
expert  and  adaptive  expert  categories,  an  accepted,  domain  specific  vocabulary  (or  jargon)  is  developed 
to  allow  efficient  communication  among  exf}erts  and  masters  in  a  given  domain.  This  is  obvious  in 
aircraft  operations  (from  flight  planning  to  air  traffic  control)  where  experts  have  developed  an 
extensive  jargon  which  is  formalized  in  the  'Tilot-Controller  Glossary"  of  the  Airman's  Information 
Manual. 


PILOT  COGNITION  &  INFORMATION  PROCESSING 

Thus  far,  we  have  characterized  the  performance  of  experts  from  a  cognitive  psychology  perspective. 
We  have  tried  to  show  that  the  development  of  expertise  relies  heavily  on  training  and  requires 
considerable  amounts  of  experience  in  a  specific  field.  Further,  experts  rely  on  a  wide  variety  of 
different  processing  skills  and  unique  problem  solving  capabilities.  As  summarized  in  Gordon  (1990): 

•  Experts  have  more  detailed,  better  organized  knowledge  structures. 

•  Experts  perceive  and  organize  problems  on  a  more  abstract  level  than  novices. 

•  Experts  perceive  problems  in  large  meaningful  patterns  related  to  the  context. 

•  Experts  are  much  faster  than  novices  because  of  their  use  of  procedural 
knowledge  and  forward  inferendng  techniques. 

These  characteristics  are  equally  applicable  to  the  expert  pilot  domain  and  all  have  been  observed  and 
documented.  We  will  now  attempt  to  show  how  the  cognitive  psychology  perspectives  and 
understanting  of  the  development  of  expertise  apply  to  pilot  development,  training  and  aeronautical 
decision  making. 

Cognitive  psychology  recognizes  three  stages  in  the  development  of  expert  problem  solving  skills 
(Anderson  1985).  These  are  cognitive,  associative  and  autonomous  .  During  the  first,  cognitive  stage, 
pilots  commit  to  memory  a  set  of  facts  relevant  to  a  desired  skill.  They  typically  rehearse  these  facts 
as  they  first  perform  the  skill.  For  example,  novice  pilots  learning  stall  recovery  will  memorize: 
recognize  the  stall,  lower  the  nose,  apply  full  power,  level  the  wings  and  minimize  altitude  loss.  In 
this  stage,  they  are  using  their  general  aeronautics  knowledge  (domain-general)  to  guide  their  solution 
to  loss  of  lift  over  one  wing,  and  solve  a  domain  specific  problem,  how  to  keep  the  aircraft  flying.  The 
problem  solving  capabilities  and  level  of  expertise  in  this  stage  are  very  basic.  Novices  spend  a  lot  of 
time  searching  and  moving  around  factual  knowledge. 

The  second,  or  associative  stage,  has  two  important  characteristics.  First,  errors  in  the  initial 
understanding  and  perfomumce  are  detected  and  gradually  eliminated.  That  is,  the  novice  pilot  learns 
to  coordinate  the  nose  drop,  power  application  and  rudder  application  for  a  smooth  stall  recovery. 
Second,  the  connections  between  the  various  elements  required  for  successful  performance  are 
strengthened.  The  pilot  does  not  sit  for  a  few  seconds  trying  to  decide  which  action  to  pxjrform  first 
after  lowering  the  nose.  Basically,  the  outcome  of  the  associative  stage  is  a  learned  procedure  for 
performing  a  desired  response  to  a  known  situation. 

The  third  cognitive  stage  occurs  when  the  problem  solving  procedures  become  faster  and  more 
automatic.  The  autonomous  stage  evolves  from  the  repeated  application  of  known  patterns  and  their 
associative  use  to  achieve  solutions.  The  use  of  declarative  knowledge  or  "verbal  mediation"  often 
disappears  during  this  stage  of  cognitive  processing,  at  least  for  some  tasks.  Expert  cognitive  process 
development  gradually  improves  in  a  specific  area  or  domain.  Ultimately,  the  skill  can  be  extended  to 
the  ability  to  respond  to  cues  not  previously  encountered  and  to  develop  new  solutions  or  production  rules 
applicable  to  novel  situations. 
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In  aviation,  training  is  highly  procedure  oriented  both  in  developing  flying  skills  (psychomotor)  and  in 
decision  making  skills  (cognitive  and  informational)  for  normal  and  emergency  operation  of  the 
aircraft.  These  procedures  and  skills  provide  the  foundation  for  the  development  of  more  sophisticated 
production  rules  (procedural  knowledge)  as  experience  is  gained.  The  novice  aviator  (100  - 1000  hours) 
develops  his  flying  and  decision  making  skills  through  1-5  years  of  experience,  "^is  experience  allows 
him  to  expand  his  procedural  knowledge  base  using  encounters  with  rea^  wv^rld  problems  and 
operational  constraints.  The  low  time  (1000  -  3000  hours)  pilot  is  at  the  second  sta,  '  cognitive  process 
development;  he  has  begun  to  develop  the  speed  and  quality  of  processing  of  the  Routine  Expert. 
Finally,  the  Expert  Pilot  (1000  -  10,000+  hours)  mainly  relics  on  his  automatic  cognitive  processing 
abilities.  Just  as  in  the  other  domains  of  sports,  games,  music,  and  medicine,  the  Expert  Pilot  has 
achieved  a  tremendous  base  of  procedural  knowledge  and  skills  applicable  to  normal  day-to-day  flying 
problems,  trained  emergencies  (such  as  an  engine  failure)  and  novel  or  untrainable  emergencies. 


TYPE  OF  PROCESSING 

Figure  1  EXPERT  PILOT  JUDGMENT  DEVELOPMENT 

Figure  1  illustrates  the  relationships  between  levels  of  pilot  experience,  types  of  knowledge  used  for 
problem  solving  and  the  three  stages  of  development  of  cognitive  processing  ability. 

As  shown  in  the  figure,  one  main  characteristic  of  the  development  of  expert  cognitive  processes  is  the 
continual  increase  in  decisional  speed  and  accuracy  as  experience  is  gained  in  a  specific  area,  e.g., 
aviation.  In  fact,  these  two  characteristics  arc  precisely  the  areas  of  decision  making  and  problem 
solving  most  affected  by  experience  and  training  or  "practice". 

To  summarize,  the  novice  or  ab  initio  pilot  responds  (cognitively)  to  stimuli  or  external  cues  based  upon 
an  understanding  of  a  complex,  declarative  knowledge  base.  His  decisions,  whether  normal  or  critical, 
are  typically  based  on  a  linear  problem  solving  approach  (some  type  of  checklist).  His  capabilities  are 
generally  limited  to  the  procedures  he  has  learned  and  expedited  by  the  use  of  rulcs-of-thumb  (or 
heuristics).  The  intermediate  pilot  is  becoming  an  associative  problem  solver.  He  has  the  capability 
for  an  enhanced  decision  making.  As  a  result  of  his  experience  and  additional  flight  training,  he  has 
the  capacity  for  more  dynamic  cognitive  processing.  At  the  associative  level,  he  stores  information  in 
terms  of  schemata  which  arc  modifiable  information  structures  based  upon  experience.  This 
"associative  pilot"  uses  pattern  recognition  and  dynamic  interrelationships  among  objects,  situations 
and  events  to  integrate  and  interpret  related  knowledge  instead  of  the  static,  linear  thinking  of  the 
novice.  This  pilot's  level  of  cognitive  processing  is  in  the  process  of  evolving  into  a  Routine  expert. 

In  addition  to  having  all  the  decision  making  skills  gained  through  experience  and  training,  the  Expert 
Pilot  is  "adaptive".  He  can  alter  his  procedures  in  real  time  (modify,  delete  or  expand).  He  can  create 
new  rules  and  patterns  based  upon  unique,  previously  unencountered  problem  characteristics.  This 
capability  to  creatively  respond  to  unique  problems  or  novel  task  demands  identifies  the  highest  level 
of  expert  pilot  cognitive  processes. 
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This  "adaptive"  capability  is  referred  to  as  "KNOWING  WHEN"  (Dreyfus  and  Dreyfus  1986).  That 
is,  the  Adaptive  Expxjrt  Pilot  can  perceive  the  necessity  to  alter  ingrained  procedures  based  upon  the 
parameters  and  dynamics  of  the  problem  or  situation  encountered.  When  necessary,  he  is  able  to  plan 
and  set  goals  required  to  accomplish  a  successful  solution.  It  is  believed  that  this  "KNOWING 
WHEN"  (an  almost  direct  perception  of  the  proper  course  of  action)  may  provide  the  key  to  the  next 
generation  of  ADM  training.  As  in  the  general  field  of  expertise,  isolating  and  quantifying  the  cues 
that  experts  use  to  either  trigger  a  routine  response  or  the  mechanism  to  adapt  remains  a  challenge. 

TRAINING  CONSIDERATIONS 

Two  issues  must  Iv  considerixl  when  teachiitg  decision  nuking  to  i*x|X'rieiued  pilots.  'I’hese  are  non¬ 
linear  decision  making  and  cognitive  dissonance. 

Non  Linear  Decision  Making:  Currently  there  are  a  large  number  of  both  competing  and  complementary 
decision  making  models  and  procedures.  A  few  of  these  either  were  aviation  developed,  modified  for 
aviation  use  or  applied  to  aviation.  These  include  the  DECIDE  model;  the  PASS  model;  and,  the  SAFE 
model.  However,  no  definitive  research  exists  which  allows  fur  the  identification  of  one  optimum 
decision  making  theory  --  cither  for  the  pilot  or  crew.  All  theories  have  positive  aspects  and 
drawbacks;  all  have  difficulty  in  meeting  all  the  unique  and  stringent  requirements  that  aviation 
imposes.  However,  a  great  deal  is  known  about  establishing  and  promoting  a  set  of  environmental 
conditions  which  foster  optimal  crew  decisional  processes  and  strategies  (Lofaro  1992). 

One  of  the  newest  areas  that  holds  promise  for  the  development  of  ADM  comes  from 
Mathematics/Artificial  Intelligence;  it  is  Chaos  theory.  All  current  models  of  ADM  are  essentially 
linear.  Some  models  have  branching  (decisional  "trees")  aspects,  but  all  involve  a  linear  series  of 
steps/ choices.  The  causal  chain  model  used  in  accident  analysis  is  an  example  of  an  essentially  linear 
view  of  a  complex  event.  It  is  true  that  a  linear  chain  of  prior  events  can  be  reconstructed  for  an 
accident.  But,  such  a  chain  is  not  sensitive  to  the  fact  that  small  changes  (in  the  environment,  in  time- 
pressure,  in  the  crow  composition,  etc.)  can  rruke  large — and  unpredictable  by  any  linear  model — 
differences  in  the  actions  and  consequences  as  time  passes. 

Since  many  ADM  models  arc  based  on  accident  analysis  (on  breaking  the  causal  chain),  once  again  the 
linear  DM  paradigm  is  used,  it  is  becoming  apparent  that,  especially  under  time  pressure,  we  make 
decisions  in  a  non-linear  fashion;  this  holds  also  for  group  decisions  (Lofaro  1991).  Another  short-fall 
of  ADM  models  based  on  accident  data  is  that  they  rely  on  an  analysis  of  one  (or  a  statistical 
representation  of  many)  prior  evcnt(s).  They  then  are  cither  so  general  as  to  not  be  helpful  in  a 
particular  situation  or  so  complex  and  specific  that  they  only  apply  to  one  situation  which  will  never 
re-occur  in  exactly  the  same  way.  Chaos  theory  deals  with  non-linear  systems  and  the  corresponding 
beginnings  of  the  realization  that  the  human  mind  does  not  typically  use  linear  steps  to  decide. 
Rather,  cognitive  processes  go  forward  and  backward,  sideward  and  into  many  layers  simultaneously. 
Insight  and  direction  for  expert  decision  making  training  may  be  available  from  this  field. 

Cognitive  Dissonance:  The  psychological  phenomena  of  cognitive  dissonance  may,  in  part,  help  to 
explain  why  some  high-lime  aviators  show  less  acceptance  of  ADM  —  as  well  as  less  altitudinal 
change  after  exposure  to  ADM  and/or  CRM  training  materials.  An  analogy  may  clarify  this  statement: 

Consider  if  you  will  the  baseball,  a  small  hard  core  wrapped  with  layers  of  varying  types  of 
twine  and  covered  with  an  outer  shell  of  stitched-on  horsehide.  The  core  is  well  protected  by 
the  layers  of  twine  and  the  cover.  The  twine  layers  themselves  are  of  different  strength  and 
they  may  be  wound  or  wrapped  with  more  or  less  tension.  The  core  can  be  considered  our  basic, 
deepest  values  and  beliefs;  the  twine  windings  are  less  tightly  held  beliefs  and  attitudes;  and, 
the  cover  is  what  holds  it  all  together  and  what  must  be  penetrated  to  access  the  interior  of  the 
baseball—  the  biases  and  slants  and  filters  by  which  we  initially  process  new  data. 

Following  this  analogy,  it  would  seem  that  the  attitudes  and  beliefs  of  high  time  aviators  are  closer  to 
the  core  and  therefore  less  amenable  to  change.  Or,  looked  at  another  way,  their  flying  habits  and 
attitudes  are  more  embedded,  therefore,  challenges  may  trigger  a  cognitive  dissonance  based 
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"protection".  The  initial  reaction  of  such  pilots  to  ADM  information  and  strategies  which  run  counter  to 
their  own  are  typical  cognitive  dissonance  mechanisms  by  which  people  do  not  change  their  attitudes 
and  beliefs  when  confronted  with  new,  unsettling  data. 

Typically,  the  person  experiencing  cognitive  dissonance  responds  by  challenging  or  rejecting  the  data 
which  has  caused  the  dissonance.  This  is  done  by  forgetting  it,  by  questioning  the  source,  and/or  by 
finding  others  of  similar  beliefs  and  attitudes.  This  serves  to  reinforce  the  original  beliefs  and 
attitudes  of  all  concerned.  This  can  be  a  partial  explanation  of  what  Dr.  Robert  Helnweich  has  found  in 
some  CRM  trainees  and  what  he  terms  the  "hoomerang  effect"  (Helmreich  1989).  Dr.  Helnu-eich  has 
found  that  some  pilots  receiving  CRM  training  not  only  resist  attitudinal  change,  but  also  experience 
either  a  hardening  or  "negative  increase"  of  their  initial  attitudes,  as  well  as  sometimes  attempting  to 
proselytize  others  in  the  OIM  class. 


CONCLUSIONS 

A  review  of  aviation  examples  where  expert  pilots  "saved  the  day"  either  in  whole  or  in  part, 
documented  that  pilot's  making  decisions  under  stress  exhibit  five  basic  characteristics  (Adams  and 
Ericcson  1992) 

•  REVERSION  TO  BASIC  AIRMANSHIP  SKILLS 

•  INSTANTANEOUS  RECALL  OF  TRAINING 

•  REASONED  APPROACH  IN  EMERGENQES 

•  POSITIVE  IN  APPROACH  &  EXPECTATIONS 

•  SELF-ASSURED  AND  OPTIMISTIC 

This  research  effort  has  identified  the  characteristics  of  expert  pilot  decision  making  and  identified 
the  differences  between  expert  and  novice  pilot  cognitive  processing  skills.  This  is  the  initial  step  in 
the  development  of  expert  pilot  decision  making  training.  However,  additional  research  is  required  in 
three  areas  to  further  our  understanding  of  the  how  the  adaptive  expert  pilot  functions  and  how  to 
train  novice  and  intermediate  pilots  this  "adaptiveness". 

The  first  area  is  to  acquire  a  better  understanding  of  the  adaptive  expert’s  perception  of  information 
and  his  decision  relat^  actions.  This  can  be  done  by  analysis  of  the  different  interpretations  of  task 
demands  between  novice  and  expert  pilot's  when  faced  with  the  same  cues  and  context,  i.  e.,  their 
sensing  and  filtering  cognitive  processes.  This  analysis  should  also  include  a  more  detailed 
examination  of  differences  between  novice  and  expert  pilot's  in  setting  goals  and  taking  action  on  the 
available  information. 

The  second  task  should  be  a  closer  examination  of  how  experienced  pilot's  have  applied  their  cognitive 
abilities  in  both  trainable  and  untrainable  emergency  situations.  This  empirical  data  base  coupled 
with  current  efforts  in  modeling  chaotic  systents  and  the  importance  of  knowing  when  to  "adapt" 
cognitive  processing  to  meet  novel  task  demands  may  provide  enough  information  to  postulate  an  initial 
expert  pilot  decision  making  model. 

The  third  area  to  explore  is  the  importance  and  impact  of  cognitive  dissonance  in  experienced  pilot 
decision  making  training.  An  analysis  of  the  possible  differences  in  the  expert  and  novice 
psychophysiological  attitudes  including  kinesthetic,  affective  and  cognitive  components  should  be 
performed.  The  importance  of  affective  components  and  characteristics  in  problem  solving  should  be 
analyzed  to  determine  what  relationships  might  exist  between  these  characteristics  and  decision 
making.  These  issues  should  be  addressed  in  expert  decision  making  training  which  provides  analogous 
affective  states  to  reinforce  the  development  of  analytical  relations  between  the  training  environment 
and  what  is  perceived  as  the  operational  environment. 

If  a  new  level  of  understanding  of  expert  pilot  cognitive  processes  in  these  three  areas  can  be  achieved, 
then  safety  could  be  further  improved  through  more  advanced,  tailored  training.  Again,  teaching 
judgment  or  decision  making  skills  (of  a  more  advanced  nature)  to  avoid  the  pitfalls  of  learning  totally 
from  the  expensive  school  of  accident/incident  experience. 
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IKTRODUCTIOM 

The  years  1991  and  1992  will  nark  the  third  watershed  In  the 
history  of  what  Is  now  temed  Crew  Resource  Management  (CRM) . 

The  first  two  watersheds  had  their  impetus  come  from  large-scale 
conferences  on  cockpit  Resource  Management  Issues  and  training 
(Cooper,  White  and  Lauber,  1979;  Orlady  and  Foushee/  1966). 
Present  efforts  are  and  will  be  driven  by  a  constellation  of 
break-throughs  In  the  understanding,  use  and  assessment  of  CRM. 

HISTORICAL  OVERVZBWi  CRM  Z  AMD  II 

In  the  late  1970s,  there  was  the  initial  recognition  that  human 
factors  such  as  crew  interaction  and  coordination,  communication/ 
leadership  styles  were  causal  factors  In  a  series  of  fatal 
commercial  crashes.  At  that  time,  several  seminal,  outstanding 
efforts  to  develop  aircrew  training  which  addressed  these  human 
factors  issues  were  developed  and  implemented  in  commercial  and 
military  aviation.  This  training  was  centered  around  what  was 
then  termed  "Cockpit  Resource  Management."  Such  efforts  and 
programs  serve  as  the  foundation  from  which  today's  CRM  springs— 
and  can  be  seen  historically  as  "CRM  I." 

During  this  same  time-frame,  and  extending  to  the  present,  NASA 
Ames  and  The  University  of  Texas  at  Austin  Instituted  an 
extensive  CRM  and  CRM  training  research  program.  This  research 
was  led  at  NASA-Ames  by  Dr.  Clay  Foushee  and  Dr*  John  Lauber  with 
Dr*  Robert  Kelmreich  of  UTA*  (NOTE:  Results  realized  by  these 
researchers  and  their  associates  generated  many  papers  and 
publications.  What  follows  is  only  a  small  sampling;  Foushee  and 
Manos,  1981;  Foushee,  1964;  Helmrelch,  1984;  Kelmreich  and 
Wilhelm,  1988;  Kelmreich,  Foushee,  Benson  and  Russini,  1986; 
Gregorich,  Kelmreich  and  Wilhelm  and  Chldester,  1989;  Lauber  and 
Foushee,  1981;  Laiiber,  1884). 

There  was  a  proliferation  of  CRM  in  a  great  variety  of  commercial 
and  military  training  programs*  The  Federal  Aviation 
Administration  (FAA)  accepted  the  value  of  CRM  training  by 
allowing  a  recurrent  training  waiver  to  be  granted  If  a  CRM 
course  was  taken.  This  eight  to  nine  year  period  (c.l979  - 
C.1988)  also  brought  out  and  clarified  many  issues  which  needed 
resolution.  Paramount  among  these  were  botr  to  identify  and 
evaluate  skills  resulting  from  CRM  training  and  how/ if  CRM  skills 
were  related  to  the  more  traditional  flight  control  ("stick  and 
rudder")  skills.  There  were,  and  still  are  to  some  degree, 
differing  schools  of  thought  on  these  issues.  This  period  can  be 
referred  to  as  "CRM  II."  As  with  "CRH  I",  it  is  a  part  of  the 
foundation  and  genesis  of  today's  work.  CRM  I  and  II  should  not 
be  viewed  as  separate  phases,  but  as  part  of  a  seamless  chain;  a 
process  which  is  still  occurring  and  parts  of  which  are,  and  will 
be,  with  us  (see  Figure  I) . 
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[ZN8ERY  FIGURE  Z  ABOUT  HERE] 


THE  PRESENT 

Currently,  (1992) ,  CRM  is  taking  great  strides.  This  is  the  tine 
of  crsw/team  resource  nanagenent;  a  tine  when  cabin  crew, 
dispatch  and  air  traffic  are  being  included  as  parts  of  the  team 
which  ensures  a  safe  and  efficient  flight.  The  four  drivers  for 
the  leap  forward  that  CBM  is  now  taking  are: 

1.  The  development  of  a  set  of  observable,  assessable 
performances  (called  "behavioral  performance  markers")  by 
Helmreich,  which  capture  the  specific  actions  that  make  up  CBM 
skills. 


2.  The  initial  demonstrations  of  What  many  had  long- 
believed,  that  CBM  skills  and  technical/ flight  control  skills  are 
interrelated  and  interdependent.  These  skills  could  be  "broken 
apart"  for  analyses,  but  in  actual  flightcrew  performance,  they 
were  both  simultaneous  and  embedded  in  each  other.  This  is 
called  "CBM  Integration"  and  is  represented  by  a  new  paradigm. 

3.  The  realii^ation  that  attention  needed  to  be  given  to 
the  development  of  training  which  integrated  the  CRM  and  the 
Flight  Comtrol/Technical  skills  and  these  skills  must  be  not  only 
taught  but  assessed  together. 

4.  The  beginnings  of  work  to  assess  a  flightcrew 's  CRM  and 
technical  skills  at  the  same  time— an  integrated  evaluation  of 
integrated  skills.  These  efforts  encompass  the  continuing  work  by 
major  airlines  in  the  design/development  of  flight  simulator 
scenarios  to  be  used  in  all  Line  Operational  Simulations  (LOS) , 
including  Line  Oriented  Flight  Training  (LOFT)  and  Line  Oriented 
Evaluation  (LOE) . 

These  four  break-throughs  have  taken  CRM  to  the  brink  of  CBM  III. 


A  cutting  Edge:  (crk)  integration 

The  integration,  and  assessment,  of  Crew  Resource  Management  and 
flight  control  skills  has  received  considerable  attention — and, 
a  fair  share  of  concern  and  skepticism— over  the  past  few  years. 
As  one  response,  in  1990  the  ATA  formed  an  air 
carrier/FAA/academe  working  group  to  deal  with  this,  and  other 
CRM  issues. 
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The  main  issues  In  doing  a  simultaneous  and  integrated  assessment 
of  CBM  and  flight  control  performance  revolve  around: 

a.  Identifying,  developing  and  validating  the 
observable/  rateable  performance  behaviors  that 
define  CRM. 

b.  Developing  a  behaviorally-anchored  scale,  or  set 
of  scales  by  which  to  assess  these  CRM  performance 
behaviors.  There  was  also  the  problem  of 
developing  a  sat  of  crew  performance  behaviors  for 
the  technical  flight  control  skills  simileu:  In 
format  to  the  CRM  performance  markers;  this  set 
would  then  be  used  in  any  attempt  at  the 
integration  with  the  CRM  behaviors. 

c.  Developing  an  analytic  paradigm  which  could  both 
identify  and  demonstrate  (what  are)  the  CRM 
performance  behaviors  embedded  in,  and  intrinsic 
to,  the  flight  control  skills  necessary  for  safe, 
efficient  missions,  such  a  paradigm  must  be  able 
to  analytically  show  where  the  integration  of  CRM 
and  flight  control  skills  occurred,  i.e.  where 
during  the  accomplishment  of  which 

maneuvers/ tasks/sub-tasks.  And,  any  model  should 
be  capable  of  dealing,  on  a  specific  level,  with 
not  only  different  aircraft  types  but  with 
different  environmental  conditions  and  with  the 
different  soP's  in  use  by  the  different  air 
carriers . 

Finally,  any  CRM  Integration  model  or  paradigm  needed  to  be  both 
operationally-oriented  and  very  accurate.  This  is  because  any  CRM 
Integration  paradigm  would  immediately  confront  a  mind-set  that 
has  evolved  in  the  development  and  **selling"  of  CRM  and  from  the 
idea  of  the  existence  of  "soft**  (as  opposed  to  "hard")  piloting 
skills. 


EZPAKDED  OVERVIEW 

Historically,  much  of  the  original  impetus  for  the  formalized 
CRM  research  and  development  came  from  analyses  of  a  series  of 
commercial  aviation  accidents  .  In  these  well-publicized,  fatal 
mishaps,  neither  aircraft  malfunction  nor  maintenance  were  the 
causal  f actor (s).  Rather,  communication,  command,  leadership  and 
other  psycho-social  factors  were  called  out  as  the  real  problems. 
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There  were  several  persons  who  had  recognized  these  issues  prior 
to  the  string  of  accidents  and  who  had  begun  the  research  and 
development  (or,  In  some  cases,  continued  their  existing  work) 
which  brought  forth  much  of  initial  CRM.  However,  almost  from 
Its  inception,  CRM  was  seen  as  somewhat  of  a  stand-alone  and/or  a 
"fix”  for  a  certain  type  of  accident,  usually  called  a  "pilot- 
error"  or,  later  a  "human  factors/performance  error"  accident, 
since  many  of  the  people  who  next  cane  into  the  field  were  from 
the  discipline  of  organizational  development,  a  perception  also 
grew  that  CRH  was  involved  only  with  soft,  sgulshy  "crow  like  a 
rooster”  activities  and  skills.  Lastly,  in  an  effort  to 
encourage  the  air  carriers  to  give  CRM  training  and  to  encourage 
line  pilots  to  take  this  training  in  a  non- jeopardy  context,  the 
FAA  began  to  grant  a  recurrency  waiver  if  CRM  training  was  taken. 
This  action  by  the  FAA  may  have  had  the  additional  effect  of 
reinforcing  a  view  that  CRM  training  was  a  separate,  stand-alone 
activity. 

In  fact,  CRM  often  was  a  stand-alone  piece  of  training  with 
either  insufficient  or  ineffective  input  from  the  pilot  community 
in  development  and  delivery— —and,  with  little,  or  no  effort  to 
relate  CRH  to  traditional  flying  skills.  A  mind-set  came  into 
place  which  incorporated  these  data. 

Many  pilots  and  researchers  with  flying  experience  knew,  almost 
intuitively,  that  CRM-type  skills  were  part  and  parcel  of  what 
good  aviators  did,  and  had  always  done  and  taught.  As  CRM  grew 
and  evolved,  a  view  also  began  to  grow  that  CRM  skills  had 
mistakenly  and  artificially  been  separated  from  the  flight 
control  skills.  By  the  early  1990 's,  with  on-going  research  and 
development  in  CRM,  along  with  the  FAA's  SFAR  on  Advanced 
Qualification  Programs  (AQP)  for  aircrew  providing  a  push,  CRM 
Integration  efforts  have  begun. 

Issues  a.  and  b.  above  have  been  extensively  worked  by 
Helmreich,  in  conjunction  with  several  major  air  carriers.  At 
this  time,  a  complete  set  of  flightcrew  CRM  performance  markers 
("CRM  behavioral  markers")  with  behaviorally-anchored  rating 
scales  have  been  developed  and  initial  research  on  them  has 
begun.  Several  air  carriers  are  investigating  the  use  of  these 
markers  In  LOFT  and  on  actual  flights. 

In  1991,  Captain  Kevin  Smith  (United  AlrLines)  and  Jan  Demuth 
(FAA  Flight  Standards)  developed  an  initial  set  of  performance 
markers  for  Technical/Flight  Control  skills.  Both  ths  CRM  and  the 
Technical  sets  of  markers  are  being  used  in  the  next  step  of  CRM 
Integration;  the  attempt  at  developing  an  analytic  paradigm  (i.e. 
issue  c) .  Smith  created  the  framework  for  a  model  which  does 
demonstrate  that  the  CRM  human  factors  skills  and  the  Technical 
flight  control  skills  are  interrelated,  interdependent  and  are 


often  siaultaneous  In  executlon”<*~that,  for  safe  and  efficient 
flight,- CRH  is  integral  to  flight  control,  and  vice-versa.  This 
model  is  called  the  Mission  Performance  Model  (MPM) .  (See  Figure 
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Members  of  an  Air  Transport  Association  (ata)  working  group  on 
CRH  proceeded  to  extend,  articulate  and  apply  the  MPM  to  actual 
flight  maneuvers,  such  as  an  engine  out  at  V^,  with  a  turn 
procedure  required  by  the  terrain.  The  MPM  seems  readily 
adaptable  to  use  in  crew  performance  analyses;  training 
evaluation  and  development;  crew  evaluator  training.  (Lofaro, 
1991;  Lofaro,  1992) .  However,  a  caveat  is  in  order:  The  MPM  is 
still  in  its  developmental  stage;  there  is  much  work  that  remains 
to  be  done  before  it  is  a  fully  articulated  and  operational 
paradigm.  Additionally,  as  Kuhn  has  pointed  out  is  the  case  with 
all  new  paradigms,  there  are  elements  of  controversy  attached  to 
the  use  and  ''value"  of  the  MPM. 
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THE  FAA'S  REVISED  CRM  ADVISORY  CIRCULAR 

Much  of  the  work  in  the  1990 's  should  be  somewhat  shaped  by  the 
revised  FAA  Crew  Resource  Management  Advisory  Circular  (AC  120- 
51A) •  This  gives  CRM  a  new  name,  "Crew  Resource  Management"  and 
incorporates  new  developments  in  CRM. 

The  revised  CRM  Advisory  circular  will  offer  a  now  definition  of 
CRM;  expansion  of  the  concept  of  CRM;  guidelines  for  building, 
implementing  and  evaluating  CRM  programs.  Much  of  the  CRM 
Advisory  Circular  revision  cones  from  an  ATA  subcoaonittee  on 
CRM/LOS  Integration  .  This  working  group  is  composed  of  airline, 
FAA  and  research  community  representatives.  The  revised  Advisory 
Circular  is  now  in  the  process  of  review  by  the  Industry  and 
approval  by  the  FAA,  with  an  expected  publication  date  of  mid  to 
late  1992. 


There  la  one  other  point  to  consider.  There  have  been,  and  still 
are,  attempts  to  define  a  single  method  for  aeronautical  decision 
making  (AOM) .  Additionally,  there  are  (juestions  as  to  whether 
CRM  encompasses  ADM,  or  vice  versa— and  whether  ADM  is  the 
desired  outcome  of  CRM  or  whether  good  CRM  results  in  good  ADM. 

ADM  meets  its  "crunch  time"  in  those  unique  situations  which  are 
highly  stressful  due  to  time  compression  and  a  lack  of 
established  procedures  which  cover  (sometimes,  even  analogously) 
the  situation.  It  would  seem  as  if  the  search  for  a  single 
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methodology  or  model  to  use  for  making  deals Iona  in  the  cockpit 
may  not  prove  feasible.  In  fact,  in  the  dynamic  cockpit 
environment,  a  single  method  or  model  of  ADM  may  be  counter¬ 
productive.  However,  as  research  and  development  on  ADM  goes 
foward,  emphasis  must  be  placed  on  identifying  and  training  those 
conditions  and  environments  which  stimulate  and  challenge  the 
crew  decisional  processes.] 


FAA'S  SFAR  56:  ADVANCED  QUALIFICATION  PROGRAM  (AQP) 

The  FAA's  SFAR  58,  signed  in  October  1990,  gives  reinforcement  to 
some  of  what  is  happening  in  CRM.  Under  AQP,  air  carriers  can 
design  and  develop  innovative  aircrew  training  (initial 
qualification;  transition;  recurrent)  programs — with  FAA  . 
approval.  CRM  and  its  evaluation  is  called  for  as  a  component  of 
Line  Oriented  Flight  Training/Line  Operational  Simulation 
(LOFT/LOS)  scenarios  used  in  some  flight  simulator  components  of 
aircrew  training.  Due  to  the  length  required  even  to  introduce 
vitally  related  work  on  the  use  of  CRM  in  LOFT/LOS  scenario 
development  and  evaluation,  this  article  will  not  attempt  to  go 
any  further  in  this  area. 


CRH  ZM  THE  FUTURE 

CRM  stands  out  as  the  most  significant  new  concept  for  aircrew 
training  and  certification  of  the  past  ten-or-so  years.  The 
unanticipated  effects  of  such  revolutions  of  going  from  "steam 
gauge"  to  "glass"  cockpits  make  one  somewhat  hesitant  about 
confidently  projecting  the  impact  of  CRM  III.  Nevertheless,  some 
things  about  CRM  III  seem  clear:  It  gives  aviation  safety  and 
aviation  human  factors  personnel  new  insights  and  new  challenges, 
as  well  as  the  potential  for  effective  solutions  to  very 
difficult  problems.  It  gives  commercial  (and  military)  aviation 
a  more  comprehensive  and  better  way  to  do  business — on  the  flight 
deck,  as  well  as  in  training. 


CRM  holds  the  promise  of  making  air  travel  even  safer  in  an  age 
of  increases  in  automation;  airspace  system  and  traffic 
complexity;  demands  on  airspace  capacity.  It  does  this  by  a  new 
focus  on,  and  understanding  of,  the  man/man  interface  and  the 
human  side  of  the  man/raachine  interface  and  interaction. 
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Decision  Maker  Expert  &  Why 

James  E.  Irving 

United  Airlines 

The  Effect  of  Trans-Cockpit 
Authority  Gradient  on  Navy 
&  Marine  Aircraft  Mishaps 

Dr.  Robert  A.  Alkov 

U.  S.  Navy 

How  Expert  Pilots  Think 

Rich  Adams 

AAC 

Methodology  for  Studying 
&  Training  Expertise 

Dr.  Anders  Ericsson 

U.  of  CO. 

Enhanced  ADM  Training 
Alternatives 

Rich  Adams 

AAC 

A  Cognitive  Model  for 

Training  Decision  Making 
in  Aircrews 

Dr.  Gary  A.  Klein 

Klein  Assoc. 

Elements  of  a  Theory  of 
Natural  Decision  Making 

Dr.  John  Bloomfield  Honeywell 

Shared  Mental  Models  in 

Crew  Decision  Making 

Dr.  Judith  Orasanu 

NASA 

Heliprops,  Embedded  ADM  to 
Reduce  Human  Error  Accidents 
in  Helicopters 

Roy  Fox 

Bell  Heli. 
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